pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
90+ messages / 6 participants
[nested] [flat]
Commits from this thread:
2022-03-12 e370f100f0 vacuumlazy.c: Standardize rel_pages terminology.
2022-03-12 73f6ec3d3c vacuumlazy.c: document vistest and OldestXmin.
2022-01-17 c702d656a2 heap pruning: Only call BufferGetBlockNumber() once.
2022-01-14 bb42bfb5cc Assert redirect pointers are sensible after heap_page_prune().
2022-01-14 18b87b201f dad1539aec Fix possible HOT corruption when RECENTLY_DEAD changes to DEAD while pruning.
* BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-10-29 07:00 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 90+ messages in thread
From: PG Bug reporting form @ 2021-10-29 07:00 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: exclusion@gmail.com
The following bug has been logged on the website:
Bug reference: 17255
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 14.0
Operating system: Ubuntu 20.04
Description:
The following scenario:
###
createdb regression
export PGDATABASE=regression
echo "
-- excerpt from inherit.sql
CREATE TABLE errtst_parent (
partid int not null,
shdata int not null,
data int NOT NULL DEFAULT 0,
CONSTRAINT shdata_small CHECK(shdata < 3)
) PARTITION BY RANGE (partid);
CREATE TABLE errtst_child_plaindef (
partid int not null,
shdata int not null,
data int NOT NULL DEFAULT 0,
CONSTRAINT shdata_small CHECK(shdata < 3),
CHECK(data < 10)
);
CREATE TABLE errtst_child_reorder (
data int NOT NULL DEFAULT 0,
shdata int not null,
partid int not null,
CONSTRAINT shdata_small CHECK(shdata < 3),
CHECK(data < 10)
);
ALTER TABLE errtst_parent ATTACH PARTITION errtst_child_plaindef FOR VALUES
FROM (10) TO (20);
ALTER TABLE errtst_parent ATTACH PARTITION errtst_child_reorder FOR VALUES
FROM (20) TO (30);
DROP TABLE errtst_parent;
" >/tmp/mini-inherit.sql
echo "
-- excerpt from vacuum.sql
CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);
SELECT pg_sleep(random()/500);
CREATE INDEX tmp_idx1 ON tmp (a);
" >/tmp/mini-vacuum.sql
echo "
VACUUM (skip_locked, index_cleanup off) pg_catalog.pg_class;
SELECT pg_sleep(random()/500);
" >/tmp/pseudo-autovacuum.sql
pgbench -n -f /tmp/mini-vacuum.sql -f /tmp/pseudo-autovacuum.sql -C -c 40 -T
600 >/dev/null 2>&1 &
pgbench -n -f /tmp/mini-inherit.sql -C -c 1 -T 600 >/dev/null 2>&1 &
wait
###
with the settings:
autovacuum=off
fsync=off
in postgresql.conf
causes the server crash:
TIME PID UID GID SIG COREFILE EXE
Fri 2021-10-29 09:37:09 MSK 383805 1000 1000 6 present
.../usr/local/pgsql/bin/postgres
real 3m20,335s
user 0m7,245s
sys 0m8,306s
with the following stack:
Core was generated by `postgres: law regression [local] CREATE INDEX
'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f8a7f97a859 in __GI_abort () at abort.c:79
#2 0x0000562dabb49700 in index_delete_sort_cmp (deltid2=<synthetic
pointer>, deltid1=<optimized out>) at heapam.c:7582
#3 index_delete_sort (delstate=0x7fff6f609f10, delstate=0x7fff6f609f10) at
heapam.c:7623
#4 heap_index_delete_tuples (rel=0x7f8a76523e08, delstate=0x7fff6f609f10)
at heapam.c:7296
#5 0x0000562dabc5519a in table_index_delete_tuples
(delstate=0x7fff6f609f10, rel=0x562dac23d6c2)
at ../../../../src/include/access/tableam.h:1327
#6 _bt_delitems_delete_check (rel=rel@entry=0x7f8a7652cc80,
buf=buf@entry=191, heapRel=heapRel@entry=0x7f8a76523e08,
delstate=delstate@entry=0x7fff6f609f10) at nbtpage.c:1541
#7 0x0000562dabc4dbe1 in _bt_simpledel_pass (maxoff=<optimized out>,
minoff=<optimized out>, newitem=<optimized out>,
ndeletable=55, deletable=0x7fff6f609f30, heapRel=0x7f8a76523e08,
buffer=191, rel=0x7f8a7652cc80)
at nbtinsert.c:2899
#8 _bt_delete_or_dedup_one_page (rel=0x7f8a7652cc80,
heapRel=0x7f8a76523e08, insertstate=0x7fff6f60a340,
simpleonly=<optimized out>, checkingunique=<optimized out>,
uniquedup=<optimized out>, indexUnchanged=false)
at nbtinsert.c:2712
#9 0x0000562dabc523f3 in _bt_findinsertloc (heapRel=0x7f8a76523e08,
stack=0x562dad8c1320, indexUnchanged=false,
checkingunique=true, insertstate=0x7fff6f60a340, rel=0x7f8a7652cc80) at
nbtinsert.c:904
#10 _bt_doinsert (rel=rel@entry=0x7f8a7652cc80,
itup=itup@entry=0x562dad8b9f20,
checkUnique=checkUnique@entry=UNIQUE_CHECK_YES,
indexUnchanged=indexUnchanged@entry=false,
heapRel=heapRel@entry=0x7f8a76523e08) at nbtinsert.c:255
#11 0x0000562dabc58451 in btinsert (rel=0x7f8a7652cc80, values=<optimized
out>, isnull=<optimized out>,
ht_ctid=0x562dad81f7d4, heapRel=0x7f8a76523e08,
checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false,
indexInfo=0x562dad8b9ad8) at nbtree.c:199
#12 0x0000562dabcc0f14 in CatalogIndexInsert
(indstate=indstate@entry=0x562dad8c0fb0,
heapTuple=heapTuple@entry=0x562dad81f7d0) at indexing.c:158
#13 0x0000562dabcc119f in CatalogTupleInsert (heapRel=0x7f8a76523e08,
tup=0x562dad81f7d0) at indexing.c:231
#14 0x0000562dabcb92a6 in InsertPgClassTuple
(pg_class_desc=pg_class_desc@entry=0x7f8a76523e08,
new_rel_desc=new_rel_desc@entry=0x7f8a7654cd28, new_rel_oid=<optimized
out>, relacl=relacl@entry=0,
reloptions=reloptions@entry=0) at heap.c:986
#15 0x0000562dabcbec8d in index_create
(heapRelation=heapRelation@entry=0x7f8a7654c6a0,
indexRelationName=indexRelationName@entry=0x562dad7f7ff8 "tmp_idx1",
indexRelationId=571743,
indexRelationId@entry=0, parentIndexRelid=parentIndexRelid@entry=0,
parentConstraintId=parentConstraintId@entry=0,
relFileNode=<optimized out>, indexInfo=<optimized out>,
indexColNames=<optimized out>,
accessMethodObjectId=<optimized out>, tableSpaceId=<optimized out>,
collationObjectId=<optimized out>,
classObjectId=<optimized out>, coloptions=<optimized out>,
reloptions=<optimized out>, flags=<optimized out>,
constr_flags=<optimized out>, allow_system_table_mods=<optimized out>,
is_internal=<optimized out>,
constraintId=<optimized out>) at index.c:968
#16 0x0000562dabd533f8 in DefineIndex (relationId=relationId@entry=571674,
stmt=stmt@entry=0x562dad7f8168,
indexRelationId=indexRelationId@entry=0,
parentIndexId=parentIndexId@entry=0,
parentConstraintId=parentConstraintId@entry=0,
is_alter_table=is_alter_table@entry=false, check_rights=true,
check_not_in_use=true, skip_build=false, quiet=false) at
indexcmds.c:1137
#17 0x0000562dabf41217 in ProcessUtilitySlow (pstate=0x562dad8c0c80,
pstmt=0x562dad7f8518,
queryString=0x562dad7f75e0 "CREATE INDEX tmp_idx1 ON tmp (a);",
context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
queryEnv=0x0, qc=0x7fff6f60b2a0, dest=<optimized out>) at
utility.c:1534
#18 0x0000562dabf3fd23 in standard_ProcessUtility (pstmt=0x562dad7f8518,
queryString=0x562dad7f75e0 "CREATE INDEX tmp_idx1 ON tmp (a);",
readOnlyTree=<optimized out>,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x562dad7f8608, qc=0x7fff6f60b2a0)
at utility.c:1066
#19 0x0000562dabf3e3f1 in PortalRunUtility
(portal=portal@entry=0x562dad85b040, pstmt=pstmt@entry=0x562dad7f8518,
isTopLevel=isTopLevel@entry=true,
setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x562dad7f8608,
qc=qc@entry=0x7fff6f60b2a0) at pquery.c:1155
#20 0x0000562dabf3e52d in PortalRunMulti
(portal=portal@entry=0x562dad85b040, isTopLevel=isTopLevel@entry=true,
setHoldSnapshot=setHoldSnapshot@entry=false,
dest=dest@entry=0x562dad7f8608, altdest=altdest@entry=0x562dad7f8608,
qc=qc@entry=0x7fff6f60b2a0) at pquery.c:1312
#21 0x0000562dabf3ebc9 in PortalRun (portal=portal@entry=0x562dad85b040,
count=count@entry=9223372036854775807,
isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true,
dest=dest@entry=0x562dad7f8608,
altdest=altdest@entry=0x562dad7f8608, qc=0x7fff6f60b2a0) at
pquery.c:788
#22 0x0000562dabf3a93b in exec_simple_query (query_string=0x562dad7f75e0
"CREATE INDEX tmp_idx1 ON tmp (a);")
at postgres.c:1214
#23 0x0000562dabf3c541 in PostgresMain (argc=argc@entry=1,
argv=argv@entry=0x7fff6f60b710, dbname=<optimized out>,
username=<optimized out>) at postgres.c:4486
#24 0x0000562dabea84bd in BackendRun (port=0x562dad81be40,
port=0x562dad81be40) at postmaster.c:4506
#25 BackendStartup (port=0x562dad81be40) at postmaster.c:4228
#26 ServerLoop () at postmaster.c:1745
#27 0x0000562dabea9461 in PostmasterMain (argc=<optimized out>,
argv=<optimized out>) at postmaster.c:1417
#28 0x0000562dabbd72e2 in main (argc=3, argv=0x562dad7f18a0) at main.c:209
Discovered while hunting to another bug related to autovacuum (unfortunately
I still can't produce the reliable reproducing script for that).
Best regards,
Alexander
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-10-29 14:55 Dmitry Dolgov <9erthalion6@gmail.com>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 90+ messages in thread
From: Dmitry Dolgov @ 2021-10-29 14:55 UTC (permalink / raw)
To: exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
> On Fri, Oct 29, 2021 at 07:00:01AM +0000, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 17255
> Logged by: Alexander Lakhin
> Email address: exclusion@gmail.com
> PostgreSQL version: 14.0
> Operating system: Ubuntu 20.04
> Description:
>
> with the following stack:
> Core was generated by `postgres: law regression [local] CREATE INDEX
> '.
> Program terminated with signal SIGABRT, Aborted.
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> #1 0x00007f8a7f97a859 in __GI_abort () at abort.c:79
> #2 0x0000562dabb49700 in index_delete_sort_cmp (deltid2=<synthetic
> pointer>, deltid1=<optimized out>) at heapam.c:7582
> #3 index_delete_sort (delstate=0x7fff6f609f10, delstate=0x7fff6f609f10) at
> heapam.c:7623
> #4 heap_index_delete_tuples (rel=0x7f8a76523e08, delstate=0x7fff6f609f10)
> at heapam.c:7296
> #5 0x0000562dabc5519a in table_index_delete_tuples
> (delstate=0x7fff6f609f10, rel=0x562dac23d6c2)
> at ../../../../src/include/access/tableam.h:1327
> #6 _bt_delitems_delete_check (rel=rel@entry=0x7f8a7652cc80,
> buf=buf@entry=191, heapRel=heapRel@entry=0x7f8a76523e08,
> delstate=delstate@entry=0x7fff6f609f10) at nbtpage.c:1541
> #7 0x0000562dabc4dbe1 in _bt_simpledel_pass (maxoff=<optimized out>,
> minoff=<optimized out>, newitem=<optimized out>,
> ndeletable=55, deletable=0x7fff6f609f30, heapRel=0x7f8a76523e08,
> buffer=191, rel=0x7f8a7652cc80)
> at nbtinsert.c:2899
> ...
>
> Discovered while hunting to another bug related to autovacuum (unfortunately
> I still can't produce the reliable reproducing script for that).
Thanks for reporting (in fact I'm impressed how many issues you've
discovered, hopefully there are at least some t-shirts "I've found X
bugs in PostgreSQL" available as a reward) and putting efforts into the
reproducing steps. I believe I've managed to reproduce at least a
similar crash with the same trace.
In my case it crashed on pg_unreachable (which is an abort, when asserts
are enabled) inside index_delete_sort_cmp. It seems like item pointers
to compare both have the same block and offset number. In the view of
the recent discussions I was thinking it could be somehow related to the
issues with duplicated TIDs, but delstate->deltids doesn't in fact have
any duplicated entries -- so not sure about that, still investigating
the core dump.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-10-30 21:39 Andres Freund <andres@anarazel.de>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 2 replies; 90+ messages in thread
From: Andres Freund @ 2021-10-30 21:39 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
Hi,
On 2021-10-29 16:55:32 +0200, Dmitry Dolgov wrote:
> > On Fri, Oct 29, 2021 at 07:00:01AM +0000, PG Bug reporting form wrote:
> > The following bug has been logged on the website:
> >
> > Bug reference: 17255
> > Logged by: Alexander Lakhin
> > Email address: exclusion@gmail.com
> > PostgreSQL version: 14.0
> > Operating system: Ubuntu 20.04
> > Description:
> >
> > with the following stack:
> > Core was generated by `postgres: law regression [local] CREATE INDEX
> > '.
> > Program terminated with signal SIGABRT, Aborted.
> > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> > 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> > (gdb) bt
> > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> > #1 0x00007f8a7f97a859 in __GI_abort () at abort.c:79
> > #2 0x0000562dabb49700 in index_delete_sort_cmp (deltid2=<synthetic
> > pointer>, deltid1=<optimized out>) at heapam.c:7582
> > #3 index_delete_sort (delstate=0x7fff6f609f10, delstate=0x7fff6f609f10) at
> > heapam.c:7623
> > #4 heap_index_delete_tuples (rel=0x7f8a76523e08, delstate=0x7fff6f609f10)
> > at heapam.c:7296
> > #5 0x0000562dabc5519a in table_index_delete_tuples
> > (delstate=0x7fff6f609f10, rel=0x562dac23d6c2)
> > at ../../../../src/include/access/tableam.h:1327
> > #6 _bt_delitems_delete_check (rel=rel@entry=0x7f8a7652cc80,
> > buf=buf@entry=191, heapRel=heapRel@entry=0x7f8a76523e08,
> > delstate=delstate@entry=0x7fff6f609f10) at nbtpage.c:1541
> > #7 0x0000562dabc4dbe1 in _bt_simpledel_pass (maxoff=<optimized out>,
> > minoff=<optimized out>, newitem=<optimized out>,
> > ndeletable=55, deletable=0x7fff6f609f30, heapRel=0x7f8a76523e08,
> > buffer=191, rel=0x7f8a7652cc80)
> > at nbtinsert.c:2899
> > ...
> >
> > Discovered while hunting to another bug related to autovacuum (unfortunately
> > I still can't produce the reliable reproducing script for that).
>
> Thanks for reporting (in fact I'm impressed how many issues you've
> discovered, hopefully there are at least some t-shirts "I've found X
> bugs in PostgreSQL" available as a reward) and putting efforts into the
> reproducing steps. I believe I've managed to reproduce at least a
> similar crash with the same trace.
>
> In my case it crashed on pg_unreachable (which is an abort, when asserts
> are enabled) inside index_delete_sort_cmp. It seems like item pointers
> to compare both have the same block and offset number. In the view of
> the recent discussions I was thinking it could be somehow related to the
> issues with duplicated TIDs, but delstate->deltids doesn't in fact have
> any duplicated entries -- so not sure about that, still investigating
> the core dump.
I suspect this is the same bug as #17245. Could you check if it's fixed by
https://www.postgresql.org/message-id/CAH2-WzkN5aESSLfK7-yrYgsXxYUi__VzG4XpZFwXm98LUtoWuQ%40mail.gma...
The crash is somewhere in pg_class, which is also manually VACUUMed by the
test, which could trigger the issue we found in the other thread. The likely
reason the loop in the repro is needed is that that'll push one of the indexes
on pg_class over the 512kb/min_parallel_index_scan_size boundary to start
using paralell vacuum.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-10-30 21:46 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-10-30 21:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Oct 30, 2021 at 2:39 PM Andres Freund <andres@anarazel.de> wrote:
> The crash is somewhere in pg_class, which is also manually VACUUMed by the
> test, which could trigger the issue we found in the other thread. The likely
> reason the loop in the repro is needed is that that'll push one of the indexes
> on pg_class over the 512kb/min_parallel_index_scan_size boundary to start
> using paralell vacuum.
Quite possible, but I can't rule out the possibility that it's
actually this other bug, since as you say this is pg_class, a system
catalog index:
https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.g...
I am in too much of a hurry right now to spend the 5 minutes it would
take to confirm either way. (Even if it's not this other bug, I needed
to remind you of its existence anyway.)
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-10-31 19:20 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Dmitry Dolgov @ 2021-10-31 19:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: exclusion@gmail.com; pgsql-bugs@lists.postgresql.org
> On Sat, Oct 30, 2021 at 02:39:48PM -0700, Andres Freund wrote:
> > In my case it crashed on pg_unreachable (which is an abort, when asserts
> > are enabled) inside index_delete_sort_cmp. It seems like item pointers
> > to compare both have the same block and offset number. In the view of
> > the recent discussions I was thinking it could be somehow related to the
> > issues with duplicated TIDs, but delstate->deltids doesn't in fact have
> > any duplicated entries -- so not sure about that, still investigating
> > the core dump.
>
> I suspect this is the same bug as #17245. Could you check if it's fixed by
> https://www.postgresql.org/message-id/CAH2-WzkN5aESSLfK7-yrYgsXxYUi__VzG4XpZFwXm98LUtoWuQ%40mail.gma...
>
> The crash is somewhere in pg_class, which is also manually VACUUMed by the
> test, which could trigger the issue we found in the other thread. The likely
> reason the loop in the repro is needed is that that'll push one of the indexes
> on pg_class over the 512kb/min_parallel_index_scan_size boundary to start
> using paralell vacuum.
I've applied both patches from Peter, the fix itself and
index-points-to-LP_UNUSED-item assertions. Now it doesn't crash on
pg_unreachable, but hits those extra assertions in the second patch:
#0 0x00007f251875f2fb in raise () from /lib64/libc.so.6
#1 0x00007f2518748ef6 in abort () from /lib64/libc.so.6
#2 0x000056387b62a4c7 in ExceptionalCondition (conditionName=0x56387b6be622 "ItemIdIsUsed(iid)", errorType=0x56387b6bc849 "FailedAssertion", fileName=0x56387b6bc928 "heapam.c", lineNumber=7467) at assert.c:69
#3 0x000056387afb4ba9 in heap_index_delete_tuples (rel=0x7f25195f8e20, delstate=0x7ffe817bdf00) at heapam.c:7467
#4 0x000056387afe4a38 in table_index_delete_tuples (rel=0x7f25195f8e20, delstate=0x7ffe817bdf00) at ../../../../src/include/access/tableam.h:1327
#5 0x000056387afe83b7 in _bt_delitems_delete_check (rel=0x7f2519601880, buf=182, heapRel=0x7f25195f8e20, delstate=0x7ffe817bdf00) at nbtpage.c:1541
#6 0x000056387afe4452 in _bt_simpledel_pass (rel=0x7f2519601880, buffer=182, heapRel=0x7f25195f8e20, deletable=0x7ffe817bdfb0, ndeletable=55, newitem=0x56387c05cfe0, minoff=1, maxoff=271) at nbtinsert.c:2896
#7 0x000056387afe3cb1 in _bt_delete_or_dedup_one_page (rel=0x7f2519601880, heapRel=0x7f25195f8e20, insertstate=0x7ffe817be3b0, simpleonly=false, checkingunique=true, uniquedup=false, indexUnchanged=false) at nbtinsert.c:2709
#8 0x000056387afdea4b in _bt_findinsertloc (rel=0x7f2519601880, insertstate=0x7ffe817be3b0, checkingunique=true, indexUnchanged=false, stack=0x56387c05d008, heapRel=0x7f25195f8e20) at nbtinsert.c:901
#9 0x000056387afdd3c7 in _bt_doinsert (rel=0x7f2519601880, itup=0x56387c05cfe0, checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false, heapRel=0x7f25195f8e20) at nbtinsert.c:255
#10 0x000056387afecfee in btinsert (rel=0x7f2519601880, values=0x7ffe817be510, isnull=0x7ffe817be4f0, ht_ctid=0x56387c05b994, heapRel=0x7f25195f8e20, checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false, indexInfo=0x56387c05cec8) at nbtree.c:199
#11 0x000056387afd7f05 in index_insert (indexRelation=0x7f2519601880, values=0x7ffe817be510, isnull=0x7ffe817be4f0, heap_t_ctid=0x56387c05b994, heapRelation=0x7f25195f8e20, checkUnique=UNIQUE_CHECK_YES, indexUnchanged=false, indexInfo=0x56387c05cec8) at indexam.c:193
#12 0x000056387b08c396 in CatalogIndexInsert (indstate=0x56387bfc0388, heapTuple=0x56387c05b990) at indexing.c:158
#13 0x000056387b08c51a in CatalogTupleInsert (heapRel=0x7f25195f8e20, tup=0x56387c05b990) at indexing.c:231
#14 0x000056387b07ed40 in InsertPgClassTuple (pg_class_desc=0x7f25195f8e20, new_rel_desc=0x7f251960fa18, new_rel_oid=957915, relacl=0, reloptions=0) at heap.c:984
#15 0x000056387b07eec6 in AddNewRelationTuple (pg_class_desc=0x7f25195f8e20, new_rel_desc=0x7f251960fa18, new_rel_oid=957915, new_type_oid=957917, reloftype=0, relowner=10, relkind=114 'r', relfrozenxid=412531, relminmxid=1, relacl=0, reloptions=0) at heap.c:1056
#16 0x000056387b07f60d in heap_create_with_catalog (relname=0x7ffe817bec60 "tmp", relnamespace=16686, reltablespace=0, relid=957915, reltypeid=0, reloftypeid=0, ownerid=10, accessmtd=2, tupdesc=0x56387bfbb6b0, cooked_constraints=0x0, relkind=114 'r', relpersistence=116 't', shared_relation=false, mapped_relation=false, oncommit=ONCOMMIT_NOOP, reloptions=0, use_user_acl=true, allow_system_table_mods=false, is_internal=false, relrewrite=0, typaddress=0x0) at heap.c:1409
#17 0x000056387b1a9bb7 in DefineRelation (stmt=0x56387bf98620, relkind=114 'r', ownerId=10, typaddress=0x0, queryString=0x56387bf97810 "CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);") at tablecmds.c:933
#18 0x000056387b47fde1 in ProcessUtilitySlow (pstate=0x56387bfb9890, pstmt=0x56387bf989d0, queryString=0x56387bf97810 "CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at utility.c:1163
#19 0x000056387b47fb3d in standard_ProcessUtility (pstmt=0x56387bf989d0, queryString=0x56387bf97810 "CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at utility.c:1066
#20 0x000056387b47eb01 in ProcessUtility (pstmt=0x56387bf989d0, queryString=0x56387bf97810 "CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at utility.c:527
#21 0x000056387b47d5e8 in PortalRunUtility (portal=0x56387bffb860, pstmt=0x56387bf989d0, isTopLevel=true, setHoldSnapshot=false, dest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at pquery.c:1155
#22 0x000056387b47d858 in PortalRunMulti (portal=0x56387bffb860, isTopLevel=true, setHoldSnapshot=false, dest=0x56387bf98ac0, altdest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at pquery.c:1312
#23 0x000056387b47ccdb in PortalRun (portal=0x56387bffb860, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x56387bf98ac0, altdest=0x56387bf98ac0, qc=0x7ffe817bf4a0) at pquery.c:788
#24 0x000056387b475fad in exec_simple_query (query_string=0x56387bf97810 "CREATE TEMPORARY TABLE tmp (a int PRIMARY KEY);") at postgres.c:1214
#25 0x000056387b47ab2b in PostgresMain (dbname=0x56387bfc3748 "regression", username=0x56387bfc3728 "erthalion") at postgres.c:4497
#26 0x000056387b3a636f in BackendRun (port=0x56387bfbb460) at postmaster.c:4560
#27 0x000056387b3a5c60 in BackendStartup (port=0x56387bfbb460) at postmaster.c:4288
#28 0x000056387b3a1e81 in ServerLoop () at postmaster.c:1801
#29 0x000056387b3a1630 in PostmasterMain (argc=3, argv=0x56387bf91ca0) at postmaster.c:1473
#30 0x000056387b297f62 in main (argc=3, argv=0x56387bf91ca0) at main.c:198
The ItemId in question:
>>> p *iid
$2 = {
lp_off = 0,
lp_flags = 0,
lp_len = 0
}
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-07 18:00 Alexander Lakhin <exclusion@gmail.com>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 1 reply; 90+ messages in thread
From: Alexander Lakhin @ 2021-11-07 18:00 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; Andres Freund <andres@anarazel.de>; Peter Geoghegan <pg@bowt.ie>; Matthias van de Meent <boekewurm+postgres@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
31.10.2021 22:20, Dmitry Dolgov wrote:
>>
>> I suspect this is the same bug as #17245. Could you check if it's fixed by
>> https://www.postgresql.org/message-id/CAH2-WzkN5aESSLfK7-yrYgsXxYUi__VzG4XpZFwXm98LUtoWuQ%40mail.gma...
>>
>> The crash is somewhere in pg_class, which is also manually VACUUMed by the
>> test, which could trigger the issue we found in the other thread. The likely
>> reason the loop in the repro is needed is that that'll push one of the indexes
>> on pg_class over the 512kb/min_parallel_index_scan_size boundary to start
>> using paralell vacuum.
> I've applied both patches from Peter, the fix itself and
> index-points-to-LP_UNUSED-item assertions. Now it doesn't crash on
> pg_unreachable, but hits those extra assertions in the second patch:
Yes, the committed fix for the bug #17245 doesn't help here.
I've also noticed that the server crash is not the only possible
outcome. You can also get unexpected errors like:
ERROR:Â relation "errtst_parent" already exists
ERROR:Â relation "tmp_idx1" already exists
ERROR:Â relation "errtst_child_plaindef" already exists
or
ERROR:Â could not open relation with OID 1033921
STATEMENT:Â DROP TABLE errtst_parent;
in the server.log (and no crash).
These strange errors and the crash inside index_delete_sort_cmp() can be
seen starting from the commit dc7420c2.
On the previous commit (b8443eae) the reproducing script completes
without a crash or errors (triple-checked).
Probably, the bug #17257 has the same root cause, but the patch [1]
applied to REL_14_STABLE (b0f6bd48) doesn't prevent the crash.
Initially I've thought that the infinite loop in vacuum is a problem
itself, so I decided to separate that one, but maybe both bugs are too
related to be discussed apart.
Best regards,
Alexander
[1]
https://www.postgresql.org/message-id/CAEze2Wj7O5tnM_U151Baxr5ObTJafwH%3D71_JEmgJV%2B6eBgjL7g%40mail...
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-08 17:28 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 90+ messages in thread
From: Dmitry Dolgov @ 2021-11-08 17:28 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Peter Geoghegan <pg@bowt.ie>; Matthias van de Meent <boekewurm+postgres@gmail.com>; pgsql-bugs@lists.postgresql.org
> On Sun, Nov 07, 2021 at 09:00:00PM +0300, Alexander Lakhin wrote:
> 31.10.2021 22:20, Dmitry Dolgov wrote:
> >>
> >> I suspect this is the same bug as #17245. Could you check if it's fixed by
> >> https://www.postgresql.org/message-id/CAH2-WzkN5aESSLfK7-yrYgsXxYUi__VzG4XpZFwXm98LUtoWuQ%40mail.gma...
> >>
> >> The crash is somewhere in pg_class, which is also manually VACUUMed by the
> >> test, which could trigger the issue we found in the other thread. The likely
> >> reason the loop in the repro is needed is that that'll push one of the indexes
> >> on pg_class over the 512kb/min_parallel_index_scan_size boundary to start
> >> using paralell vacuum.
> > I've applied both patches from Peter, the fix itself and
> > index-points-to-LP_UNUSED-item assertions. Now it doesn't crash on
> > pg_unreachable, but hits those extra assertions in the second patch:
> Yes, the committed fix for the bug #17245 doesn't help here.
> I've also noticed that the server crash is not the only possible
> outcome. You can also get unexpected errors like:
> ERROR:Â relation "errtst_parent" already exists
> ERROR:Â relation "tmp_idx1" already exists
> ERROR:Â relation "errtst_child_plaindef" already exists
> or
> ERROR:Â could not open relation with OID 1033921
> STATEMENT:Â DROP TABLE errtst_parent;
> in the server.log (and no crash).
Interesting, I don't think I've observed those errors. In fact after the
recent changes (I've compiled here from 39a31056) around assertion logic
and index_delete_check_htid now I'm getting another type of crashes
using your scripts. This time heap_page_prune_execute stumbles upon a
non heap-only tuple trying to update unused line pointers:
#0 0x00007f0dfce072fb in raise () from /lib64/libc.so.6
#1 0x00007f0dfcdf0ef6 in abort () from /lib64/libc.so.6
#2 0x000055a66a87db05 in ExceptionalCondition (conditionName=0x55a66a914610 "HeapTupleHeaderIsHeapOnly(htup)", errorType=0x55a66a91419c "FailedAssertion", fileName=0x55a66a914190 "pruneheap.c", lineNumber=961) at assert.c:69
#3 0x000055a66a21b83a in heap_page_prune_execute (buffer=138, redirected=0x7fffac638994, nredirected=0, nowdead=0x7fffac638e20, ndead=12, nowunused=0x7fffac639066, nunused=1) at pruneheap.c:961
#4 0x000055a66a219d82 in heap_page_prune (relation=0x7f0dfdca0e20, buffer=138, vistest=0x55a66addf140 <GlobalVisCatalogRels>, old_snap_xmin=0, old_snap_ts=0, report_stats=true, off_loc=0x0) at pruneheap.c:295
The relation in question is pg_class, and htup apparently has
e.g. HEAP_KEYS_UPDATED, but no HEAP_ONLY_TUPLE flags set:
>>> p *htup
$1 = {
t_choice = {
t_heap = {
t_xmin = 661929,
t_xmax = 662015,
t_field3 = {
t_cid = 0,
t_xvac = 0
}
},
t_datum = {
datum_len_ = 661929,
datum_typmod = 662015,
datum_typeid = 0
}
},
t_ctid = {
ip_blkid = {
bi_hi = 0,
bi_lo = 2004
},
ip_posid = 128
},
t_infomask2 = 8225,
t_infomask = 1281,
t_hoff = 32 ' ',
t_bits = 0x7f0de7080ee7 "\377\377\377?"
}
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-08 18:32 Peter Geoghegan <pg@bowt.ie>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-08 18:32 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Andres Freund <andres@anarazel.de>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Nov 8, 2021 at 9:28 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
> Interesting, I don't think I've observed those errors. In fact after the
> recent changes (I've compiled here from 39a31056) around assertion logic
> and index_delete_check_htid now I'm getting another type of crashes
> using your scripts. This time heap_page_prune_execute stumbles upon a
> non heap-only tuple trying to update unused line pointers:
It looks like the new heap_page_prune_execute() assertions catch the
same problem earlier. It's hard to not suspect the code in pruneheap.c
itself. Whatever else may have happened, the code in pruneheap.c ought
to not even try to set a non-heap-only tuple item to LP_UNUSED. ISTM
that it should explicitly look out for and avoid doing that.
Offhand, I wonder if we just need to have an additional check in
heap_prune_chain(), which is like the existing "If we find a redirect
somewhere else, stop --- it must not be same chain" handling used for
LP_REDIRECT items that aren't at the start of a HOT chain:
diff --git a/src/backend/access/heap/pruneheap.c
b/src/backend/access/heap/pruneheap.c
index c7331d810..3c72cdf67 100644
inline patch 1: src/backend/access/heap/pruneheap.c (+0/-0 lines) download inline-1.patch
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -646,6 +646,9 @@ heap_prune_chain(Buffer buffer, OffsetNumber
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-09 15:02 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Dmitry Dolgov @ 2021-11-09 15:02 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Andres Freund <andres@anarazel.de>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On Mon, Nov 08, 2021 at 10:32:39AM -0800, Peter Geoghegan wrote:
> On Mon, Nov 8, 2021 at 9:28 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
> > Interesting, I don't think I've observed those errors. In fact after the
> > recent changes (I've compiled here from 39a31056) around assertion logic
> > and index_delete_check_htid now I'm getting another type of crashes
> > using your scripts. This time heap_page_prune_execute stumbles upon a
> > non heap-only tuple trying to update unused line pointers:
>
> It looks like the new heap_page_prune_execute() assertions catch the
> same problem earlier. It's hard to not suspect the code in pruneheap.c
> itself. Whatever else may have happened, the code in pruneheap.c ought
> to not even try to set a non-heap-only tuple item to LP_UNUSED. ISTM
> that it should explicitly look out for and avoid doing that.
>
> Offhand, I wonder if we just need to have an additional check in
> heap_prune_chain(), which is like the existing "If we find a redirect
> somewhere else, stop --- it must not be same chain" handling used for
> LP_REDIRECT items that aren't at the start of a HOT chain:
Yes, adding such condition works in this case, no non-heap-only tuples
were recorded as unused in heap_prune_chain, and nothing else popped up
afterwards. But now after a couple of runs I could also reproduce (at
least partially) what Alexander was talking about:
ERROR: could not open relation with OID 1056321
Not sure yet where is it coming from.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-09 17:51 Peter Geoghegan <pg@bowt.ie>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-09 17:51 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Andres Freund <andres@anarazel.de>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Nov 9, 2021 at 7:01 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
> Yes, adding such condition works in this case, no non-heap-only tuples
> were recorded as unused in heap_prune_chain, and nothing else popped up
> afterwards. But now after a couple of runs I could also reproduce (at
> least partially) what Alexander was talking about:
>
> ERROR: could not open relation with OID 1056321
I've seen that too.
> Not sure yet where is it coming from.
I think that the additional check that I sketched (in
heap_prune_chain()) is protective in that it prevents a bad situation
from becoming even worse. At the same time it doesn't actually fix
anything.
I've discussed this privately with Andres -- expect more from him
soon. I came up with more sophisticated instrumentation (better
assertions, really) that shows that the problem begins in VACUUM, not
opportunistic pruning (certainly with the test case we have).
The real problem (identified by Andres) seems to be that pruning feels
entitled to corrupt HOT chains by making an existing LP_REDIRECT
continue to point to a DEAD item that it marks LP_UNUSED. That's never
supposed to happen. This seems to occur in the aborted heap-only tuple
"If the tuple is DEAD and doesn't chain to anything else, mark it
unused immediately" code path at the top of heap_prune_chain(). That
code path seems wonky, despite not having changed much in many years.
This wonky heap_prune_chain() code cares about whether the
to-be-set-LP_UNUSED heap-only tuple item chains to other items -- it's
only really needed for aborted tuples, but doesn't discriminate
between aborted tuples and other kinds of DEAD tuples. It doesn't seem
to get all the details right. In particular, it doesn't account for
the fact that it's not okay to break a HOT chain between the root
LP_REDIRECT item and the first heap-only tuple. It's only okay to do
that between two heap-only tuples.
HOT chain traversal code (in places like heap_hot_search_buffer())
knows how to deal with broken HOT chains when the breakage occurs
between two heap-only tuples, but not in this other LP_REDIRECT case.
It's just not possible for HOT chain traversal code to deal with that,
because there is not enough information in the LP_REDIRECT (just a
link to another item, no xmin or xmax) to validate anything on the
fly. It's pretty clear that we need to specifically make sure that
LP_REDIRECT items always point to something sensible. Anything less
risks causing confusion about which HOT chain lives at which TID.
Obviously we were quite right to suspect that there wasn't enough
rigor around the HOT chain invariants. Wasn't specifically expecting
it to help with this bug.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-09 23:31 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-09 23:31 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Andres Freund <andres@anarazel.de>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Nov 9, 2021 at 9:51 AM Peter Geoghegan <pg@bowt.ie> wrote:
> I've discussed this privately with Andres -- expect more from him
> soon. I came up with more sophisticated instrumentation (better
> assertions, really) that shows that the problem begins in VACUUM, not
> opportunistic pruning (certainly with the test case we have).
Attached is a WIP fix for the bug. The idea here is to follow all HOT
chains in an initial pass over the page, while even following LIVE
heap-only tuples. Any heap-only tuples that we don't determine are
part of some valid HOT chain (following an initial pass over the whole
heap page) will now be processed in a second pass over the page. We
expect (and assert) that these "disconnected" heap-only tuples will
all be either DEAD or RECENTLY_DEAD. We treat them as DEAD either way,
on the grounds that they must be from an aborted xact in any case.
Note that we sometimes do something very similar already -- we can
sometimes consider some tuples from a HOT chain DEAD, even though
they're RECENTLY_DEAD (provided a later tuple from the chain really is
DEAD).
The patch also has more detailed assertions inside heap_page_prune().
These should catch any HOT chain invariant violations at just about
the earliest opportunity, at least when assertions are enabled.
Especially because we're now following every HOT chain from beginning
to end now, even when we already know that there are no more
DEAD/RECENTLY_DEAD tuples in the chain to be found.
I'm not sure why this seems to have become more of a problem following
the snapshot scalability work from Andres -- Alexander mentioned that
commit dc7420c2 looked like it was the source of the problem here, but
I can't see any reason why that might be true (even though I accept
that it might well *appear* to be true). I believe Andres has some
theory on that, but I don't know the details myself. AFAICT, this is a
live bug on all supported versions. We simply weren't being careful
enough about breaking the invariant that an LP_REDIRECT can only point
to a valid heap-only tuple. The really surprising thing here is that
it took this long for it to visibly break.
--
Peter Geoghegan
Attachments:
v1-0001-Fix-aborted-HOT-update-bug-in-heap-pruning.patch (application/octet-stream, 12.2K) download
From c6996e3743d75df11d3d06508c4f532e61f9f8e6 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 8 Nov 2021 22:58:14 -0800
Subject: [PATCH v1] Fix aborted HOT update bug in heap pruning.
---
src/backend/access/heap/pruneheap.c | 201 +++++++++++++++++++---------
1 file changed, 135 insertions(+), 66 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c7331d810..7cad94705 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -58,12 +58,16 @@ typedef struct
OffsetNumber nowunused[MaxHeapTuplesPerPage];
/* marked[i] is true if item i is entered in one of the above arrays */
bool marked[MaxHeapTuplesPerPage + 1];
+ /* inhotchain[i] is true if item i is in valid HOT chain */
+ bool inhotchain[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate);
+static int heap_prune_heaponly(Buffer buffer, OffsetNumber heaponlyoffnum,
+ PruneState *prstate);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -249,6 +253,7 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.latestRemovedXid = InvalidTransactionId;
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
memset(prstate.marked, 0, sizeof(prstate.marked));
+ memset(prstate.inhotchain, 0, sizeof(prstate.inhotchain));
/* Scan the page */
maxoff = PageGetMaxOffsetNumber(page);
@@ -278,6 +283,35 @@ heap_page_prune(Relation relation, Buffer buffer,
ndeleted += heap_prune_chain(buffer, offnum, &prstate);
}
+ /*
+ * Scan the page again, to pick up any heap-only tuples that were not
+ * recognized as part of some HOT chain during our first pass
+ */
+ for (offnum = FirstOffsetNumber;
+ offnum <= maxoff;
+ offnum = OffsetNumberNext(offnum))
+ {
+ ItemId itemid;
+
+ /* Ignore items already processed as part of an earlier chain */
+ if (prstate.inhotchain[offnum])
+ continue;
+ Assert(!prstate.marked[offnum]);
+ itemid = PageGetItemId(page, offnum);
+ if (!ItemIdIsNormal(itemid))
+ continue;
+
+ /*
+ * Set the offset number so that we can display it along with any
+ * error that occurred while processing this tuple.
+ */
+ if (off_loc)
+ *off_loc = offnum;
+
+ /* Process this item */
+ ndeleted += heap_prune_heaponly(buffer, offnum, &prstate);
+ }
+
/* Clear the offset information once we have processed the given page. */
if (off_loc)
*off_loc = InvalidOffsetNumber;
@@ -521,73 +555,48 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
Page dp = (Page) BufferGetPage(buffer);
TransactionId priorXmax = InvalidTransactionId;
ItemId rootlp;
- HeapTupleHeader htup;
OffsetNumber latestdead = InvalidOffsetNumber,
maxoff = PageGetMaxOffsetNumber(dp),
offnum;
OffsetNumber chainitems[MaxHeapTuplesPerPage];
- int nchain = 0,
- i;
- HeapTupleData tup;
-
- tup.t_tableOid = RelationGetRelid(prstate->rel);
+ int nchain = 0;
+ bool startswithredirect PG_USED_FOR_ASSERTS_ONLY = false;
rootlp = PageGetItemId(dp, rootoffnum);
- /*
- * If it's a heap-only tuple, then it is not the start of a HOT chain.
- */
if (ItemIdIsNormal(rootlp))
{
+ HeapTupleHeader htup;
+
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
- tup.t_data = htup;
- tup.t_len = ItemIdGetLength(rootlp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), rootoffnum);
-
+ /*
+ * If it's a heap-only tuple, then it is not the start of a HOT chain.
+ * We should process it by finding it through its HOT chain (by a
+ * later call that starts at the HOT chain's root item).
+ *
+ * Sometimes that'll never happen, because the heap-only tuple will be
+ * disconnected from its HOT chain following a HOT update whose xact
+ * aborts. If that turns out to be the case for this heap-only tuple,
+ * we'll process it in heap_prune_heaponly() during our second pass
+ * over the page.
+ */
if (HeapTupleHeaderIsHeapOnly(htup))
- {
- /*
- * If the tuple is DEAD and doesn't chain to anything else, mark
- * it unused immediately. (If it does chain, we can only remove
- * it as part of pruning its chain.)
- *
- * We need this primarily to handle aborted HOT updates, that is,
- * XMIN_INVALID heap-only tuples. Those might not be linked to by
- * any chain, since the parent tuple might be re-updated before
- * any pruning occurs. So we have to be able to reap them
- * separately from chain-pruning. (Note that
- * HeapTupleHeaderIsHotUpdated will never return true for an
- * XMIN_INVALID tuple, so this code will work even when there were
- * sequential updates within the aborted transaction.)
- *
- * Note that we might first arrive at a dead heap-only tuple
- * either here or while following a chain below. Whichever path
- * gets there first will mark the tuple unused.
- */
- if (heap_prune_satisfies_vacuum(prstate, &tup, buffer)
- == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
- {
- heap_prune_record_unused(prstate, rootoffnum);
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- ndeleted++;
- }
-
- /* Nothing more to do */
- return ndeleted;
- }
+ return 0;
}
/* Start from the root tuple */
+ Assert(!prstate->inhotchain[rootoffnum]);
+ prstate->inhotchain[rootoffnum] = true;
offnum = rootoffnum;
/* while not end of the chain */
for (;;)
{
ItemId lp;
- bool tupdead,
- recent_dead;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ bool tupdead;
/* Sanity check (pure paranoia) */
if (offnum < FirstOffsetNumber)
@@ -606,10 +615,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
lp = PageGetItemId(dp, offnum);
- /* Unused item obviously isn't part of the chain */
- if (!ItemIdIsUsed(lp))
- break;
-
/*
* If we are looking at the redirected root line pointer, jump to the
* first normal tuple in the chain. If we find a redirect somewhere
@@ -619,24 +624,26 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
{
if (nchain > 0)
break; /* not at start of chain */
+ startswithredirect = true;
chainitems[nchain++] = offnum;
offnum = ItemIdGetRedirect(rootlp);
continue;
}
- /*
- * Likewise, a dead line pointer can't be part of the chain. (We
- * already eliminated the case of dead root tuple outside this
- * function.)
- */
- if (ItemIdIsDead(lp))
+ /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
+ if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
+ {
+ /* An LP_REDIRECT cannot point directly to an LP_UNUSED */
+ Assert(!startswithredirect || nchain > 1);
break;
+ }
Assert(ItemIdIsNormal(lp));
htup = (HeapTupleHeader) PageGetItem(dp, lp);
- tup.t_data = htup;
tup.t_len = ItemIdGetLength(lp);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), offnum);
/*
@@ -644,17 +651,24 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (TransactionIdIsValid(priorXmax) &&
!TransactionIdEquals(HeapTupleHeaderGetXmin(htup), priorXmax))
+ {
+ Assert(!startswithredirect || nchain > 1);
break;
+ }
+
+ Assert(nchain == 0 || HeapTupleHeaderIsHeapOnly(htup));
+ Assert(nchain == 0 || !prstate->inhotchain[offnum]);
/*
* OK, this tuple is indeed a member of the chain.
*/
chainitems[nchain++] = offnum;
+ prstate->inhotchain[offnum] = true;
/*
* Check tuple's visibility status.
*/
- tupdead = recent_dead = false;
+ tupdead = false;
switch (heap_prune_satisfies_vacuum(prstate, &tup, buffer))
{
@@ -663,7 +677,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
case HEAPTUPLE_RECENTLY_DEAD:
- recent_dead = true;
/*
* This tuple may soon become DEAD. Update the hint field so
@@ -679,6 +692,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* This tuple may soon become DEAD. Update the hint field so
* that the page is reconsidered for pruning in future.
*/
+ Assert(!tupdead);
heap_prune_record_prunable(prstate,
HeapTupleHeaderGetUpdateXid(htup));
break;
@@ -692,6 +706,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* But we don't. See related decisions about when to mark the
* page prunable in heapam.c.
*/
+ Assert(!tupdead);
break;
default:
@@ -700,11 +715,14 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * Remember the last DEAD tuple seen. We will advance past
- * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
- * but we can't advance past anything else. We have to make sure that
- * we don't miss any DEAD tuples, since DEAD tuples that still have
- * tuple storage after pruning will confuse VACUUM.
+ * Remember the last DEAD tuple seen. We will advance past all other
+ * tuples (even LIVE tuples) so that we have a clear picture of which
+ * heap-only tuples are known to be part of a valid HOT chain.
+ *
+ * Note that we expect to sometimes find a DEAD tuple after a
+ * RECENTLY_DEAD tuple. When this happens, the RECENTLY_DEAD tuple
+ * will be treated as DEAD. But there has to be a later DEAD tuple
+ * for that to happen.
*/
if (tupdead)
{
@@ -712,8 +730,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
}
- else if (!recent_dead)
- break;
/*
* If the tuple is not HOT-updated, then we are at the end of this
@@ -741,6 +757,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (OffsetNumberIsValid(latestdead))
{
+ int i;
+
/*
* Mark as unused each intermediate item that we are able to remove
* from the chain.
@@ -787,6 +805,57 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
return ndeleted;
}
+/*
+ * Handle disconnected heap-only tuples during second pass over page. We
+ * always expect to mark these tuples as DEAD here.
+ *
+ * We need this to handle aborted HOT updates, that is, XMIN_INVALID heap-only
+ * tuples. Even then, we only do this for heap-only tuples that are not
+ * locatable in any HOT chain. This happens when the parent tuple was
+ * re-updated before any pruning took place. We have to be able to reap them
+ * separately from chain-pruning.
+ *
+ * We have to make sure that we don't miss any DEAD tuples, since DEAD tuples
+ * that still have tuple storage after pruning will confuse VACUUM.
+*/
+static int
+heap_prune_heaponly(Buffer buffer, OffsetNumber heaponlyoffnum,
+ PruneState *prstate)
+{
+ Page dp = (Page) BufferGetPage(buffer);
+ ItemId lp;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ HTSV_Result res;
+
+ lp = PageGetItemId(dp, heaponlyoffnum);
+ Assert(ItemIdIsNormal(lp));
+ htup = (HeapTupleHeader) PageGetItem(dp, lp);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+ tup.t_len = ItemIdGetLength(lp);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), heaponlyoffnum);
+
+ /*
+ * We expect that disconnected heap-only tuples must be from aborted
+ * transactions. Any RECENTLY_DEAD tuples we see here are really DEAD,
+ * but the heap_prune_satisfies_vacuum test is too coarse to detect it.
+ */
+ res = heap_prune_satisfies_vacuum(prstate, &tup, buffer);
+ if (res == HEAPTUPLE_DEAD || res == HEAPTUPLE_RECENTLY_DEAD)
+ {
+ heap_prune_record_unused(prstate, heaponlyoffnum);
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ return 1;
+ }
+ else
+ Assert(false);
+
+ return 0;
+}
+
/* Record lowest soon-prunable XID */
static void
heap_prune_record_prunable(PruneState *prstate, TransactionId xid)
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-10 03:07 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-10 03:07 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Alexander Lakhin <exclusion@gmail.com>; Andres Freund <andres@anarazel.de>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Tue, Nov 9, 2021 at 3:31 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Attached is a WIP fix for the bug. The idea here is to follow all HOT
> chains in an initial pass over the page, while even following LIVE
> heap-only tuples. Any heap-only tuples that we don't determine are
> part of some valid HOT chain (following an initial pass over the whole
> heap page) will now be processed in a second pass over the page.
I realized that I could easily go further than in v1, and totally get
rid of the "marked" array (which tracks whether we have decided to
mark an item as LP_DEAD/LP_UNUSED/a new LP_REDIRECT/newly pointed to
by another LP_REDIRECT). In my v1 from earlier today we already had an
array that records whether or not each item is part of any known valid
chain, which is strictly better than knowing whether or not they were
"marked" earlier. So why bother with the "marked" array at all, even
for assertions? It is less robust (not to mention less efficient) than
just using the new "fromvalidchain" array.
Attached is v2, which gets rid of the "marked" array as described. It
also has better worked out comments and assertions. The patch has
stood up to a fair amount of stress-testing. I repeated Alexander's
original test case for over an hour with this. Getting the test case
to cause an assertion failure would usually take about 5 minutes
without any fix.
I have yet to do any work on validating the performance of this patch,
though that definitely needs to happen.
Anybody have any thoughts on how far this should be backpatched? We'll
probably need to do that for Postgres 14. Less sure about other
branches, which haven't been directly demonstrated to be affected by
the bug so far. Haven't tried to break earlier branches with
Alexander's test case, though I will note again that Alexander
couldn't do that when he tried.
--
Peter Geoghegan
Attachments:
v2-0001-Fix-aborted-HOT-update-bug-in-heap-pruning.patch (application/octet-stream, 17.9K) download
From a82c9aee92a90bd7a33f861103b5dcb9ca39d56a Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 8 Nov 2021 22:58:14 -0800
Subject: [PATCH v2] Fix aborted HOT update bug in heap pruning.
Author: Peter Geoghegan <pg@bowt.ie>
Reported-By: Alexander Lakhin <exclusion@gmail.com>
Diagnosed-By: Andres Freund <andres@anarazel.de>
Bug: #17255
Discussion: https://postgr.es/m/17255-14c0ac58d0f9b583@postgresql.org
Backpatch: ???
---
src/backend/access/heap/pruneheap.c | 290 ++++++++++++++++++----------
1 file changed, 187 insertions(+), 103 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index c7331d810..31fa4d2a3 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -31,6 +31,7 @@
typedef struct
{
Relation rel;
+ BlockNumber targetblkno;
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
@@ -56,14 +57,16 @@ typedef struct
OffsetNumber redirected[MaxHeapTuplesPerPage * 2];
OffsetNumber nowdead[MaxHeapTuplesPerPage];
OffsetNumber nowunused[MaxHeapTuplesPerPage];
- /* marked[i] is true if item i is entered in one of the above arrays */
- bool marked[MaxHeapTuplesPerPage + 1];
+ /* fromvalidchain[i] is true if item i is from a known valid HOT chain */
+ bool fromvalidchain[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate);
+static int heap_prune_heaponly(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -242,13 +245,14 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.targetblkno = BufferGetBlockNumber(buffer);
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
prstate.latestRemovedXid = InvalidTransactionId;
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
- memset(prstate.marked, 0, sizeof(prstate.marked));
+ memset(prstate.fromvalidchain, 0, sizeof(prstate.fromvalidchain));
/* Scan the page */
maxoff = PageGetMaxOffsetNumber(page);
@@ -256,10 +260,8 @@ heap_page_prune(Relation relation, Buffer buffer,
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- ItemId itemid;
-
/* Ignore items already processed as part of an earlier chain */
- if (prstate.marked[offnum])
+ if (prstate.fromvalidchain[offnum])
continue;
/*
@@ -269,15 +271,29 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- /* Nothing to do if slot is empty or already dead */
- itemid = PageGetItemId(page, offnum);
- if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
- continue;
-
/* Process this item or chain of items */
ndeleted += heap_prune_chain(buffer, offnum, &prstate);
}
+ /*
+ * Scan the page again, processing any heap-only tuples that we now know
+ * are not part of any valid HOT chain
+ */
+ for (offnum = FirstOffsetNumber;
+ offnum <= maxoff;
+ offnum = OffsetNumberNext(offnum))
+ {
+ /* Ignore items already processed as part of a known valid chain */
+ if (prstate.fromvalidchain[offnum])
+ continue;
+
+ if (off_loc)
+ *off_loc = offnum;
+
+ /* Process this disconnected heap-only tuple */
+ ndeleted += heap_prune_heaponly(buffer, offnum, &prstate);
+ }
+
/* Clear the offset information once we have processed the given page. */
if (off_loc)
*off_loc = InvalidOffsetNumber;
@@ -383,19 +399,8 @@ heap_page_prune(Relation relation, Buffer buffer,
pgstat_update_heap_dead_tuples(relation, ndeleted - prstate.ndead);
/*
- * XXX Should we update the FSM information of this page ?
- *
- * There are two schools of thought here. We may not want to update FSM
- * information so that the page is not used for unrelated UPDATEs/INSERTs
- * and any free space in this page will remain available for further
- * UPDATEs in *this* page, thus improving chances for doing HOT updates.
- *
- * But for a large table and where a page does not receive further UPDATEs
- * for a long time, we might waste this space by not updating the FSM
- * information. The relation may get extended and fragmented further.
- *
- * One possibility is to leave "fillfactor" worth of space in this page
- * and update FSM with the remaining space.
+ * Don't update the FSM information on this page. We leave it up to our
+ * caller to decide what to do about that.
*/
return ndeleted;
@@ -521,73 +526,70 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
Page dp = (Page) BufferGetPage(buffer);
TransactionId priorXmax = InvalidTransactionId;
ItemId rootlp;
- HeapTupleHeader htup;
OffsetNumber latestdead = InvalidOffsetNumber,
maxoff = PageGetMaxOffsetNumber(dp),
offnum;
OffsetNumber chainitems[MaxHeapTuplesPerPage];
- int nchain = 0,
- i;
- HeapTupleData tup;
-
- tup.t_tableOid = RelationGetRelid(prstate->rel);
+ int nchain;
+ bool rootisredirect PG_USED_FOR_ASSERTS_ONLY = false;
rootlp = PageGetItemId(dp, rootoffnum);
- /*
- * If it's a heap-only tuple, then it is not the start of a HOT chain.
- */
if (ItemIdIsNormal(rootlp))
{
+ HeapTupleHeader htup;
+
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
- tup.t_data = htup;
- tup.t_len = ItemIdGetLength(rootlp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), rootoffnum);
-
+ /*
+ * If it's a heap-only tuple, then it is not the start of a HOT chain.
+ * We should process it by finding it through its HOT chain (during a
+ * later call that starts at the HOT chain's root item).
+ *
+ * Sometimes that won't happen because the heap-only tuple will be
+ * disconnected from its HOT chain following a HOT update whose xact
+ * aborts. These heap-only tuples are processed during a later call
+ * to heap_prune_heaponly() instead.
+ */
if (HeapTupleHeaderIsHeapOnly(htup))
- {
- /*
- * If the tuple is DEAD and doesn't chain to anything else, mark
- * it unused immediately. (If it does chain, we can only remove
- * it as part of pruning its chain.)
- *
- * We need this primarily to handle aborted HOT updates, that is,
- * XMIN_INVALID heap-only tuples. Those might not be linked to by
- * any chain, since the parent tuple might be re-updated before
- * any pruning occurs. So we have to be able to reap them
- * separately from chain-pruning. (Note that
- * HeapTupleHeaderIsHotUpdated will never return true for an
- * XMIN_INVALID tuple, so this code will work even when there were
- * sequential updates within the aborted transaction.)
- *
- * Note that we might first arrive at a dead heap-only tuple
- * either here or while following a chain below. Whichever path
- * gets there first will mark the tuple unused.
- */
- if (heap_prune_satisfies_vacuum(prstate, &tup, buffer)
- == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
- {
- heap_prune_record_unused(prstate, rootoffnum);
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- ndeleted++;
- }
+ return 0;
+ }
+ else if (!ItemIdIsRedirected(rootlp))
+ {
+ /*
+ * Nothing to do if slot cannot possibly be valid root item of HOT
+ * chain or a simple heap tuple
+ */
+ Assert(ItemIdIsDead(rootlp) || !ItemIdIsUsed(rootlp));
+ prstate->fromvalidchain[rootoffnum] = true;
- /* Nothing more to do */
- return ndeleted;
- }
+ return 0;
+ }
+ else
+ {
+ /*
+ * Remember that root item is an LP_REDIRECT item, not a plain
+ * (not-heap-only) tuple item
+ */
+ rootisredirect = true;
}
- /* Start from the root tuple */
+ /*
+ * Start from the root item. Mark it as valid up front, since root items
+ * are always processed up front, in first pass over page.
+ */
+ Assert(!prstate->fromvalidchain[rootoffnum]);
+ prstate->fromvalidchain[rootoffnum] = true;
offnum = rootoffnum;
+ nchain = 0;
/* while not end of the chain */
for (;;)
{
ItemId lp;
- bool tupdead,
- recent_dead;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ bool tupdead;
/* Sanity check (pure paranoia) */
if (offnum < FirstOffsetNumber)
@@ -601,15 +603,11 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/* If item is already processed, stop --- it must not be same chain */
- if (prstate->marked[offnum])
+ if (nchain != 0 && prstate->fromvalidchain[offnum])
break;
lp = PageGetItemId(dp, offnum);
- /* Unused item obviously isn't part of the chain */
- if (!ItemIdIsUsed(lp))
- break;
-
/*
* If we are looking at the redirected root line pointer, jump to the
* first normal tuple in the chain. If we find a redirect somewhere
@@ -619,42 +617,63 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
{
if (nchain > 0)
break; /* not at start of chain */
+ Assert(rootisredirect);
chainitems[nchain++] = offnum;
offnum = ItemIdGetRedirect(rootlp);
continue;
}
- /*
- * Likewise, a dead line pointer can't be part of the chain. (We
- * already eliminated the case of dead root tuple outside this
- * function.)
- */
- if (ItemIdIsDead(lp))
+ /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
+ if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
+ {
+ /*
+ * XXX What if we just came from root item, which is a plain heap
+ * tuple? Do we need assert that notices when we reach an LP_DEAD
+ * or LP_UNUSED item having started from such a root item?
+ */
+ Assert(!rootisredirect || nchain > 1);
break;
+ }
Assert(ItemIdIsNormal(lp));
htup = (HeapTupleHeader) PageGetItem(dp, lp);
- tup.t_data = htup;
tup.t_len = ItemIdGetLength(lp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), offnum);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
/*
* Check the tuple XMIN against prior XMAX, if any
*/
if (TransactionIdIsValid(priorXmax) &&
!TransactionIdEquals(HeapTupleHeaderGetXmin(htup), priorXmax))
+ {
+ Assert(!rootisredirect || nchain > 1);
break;
+ }
/*
- * OK, this tuple is indeed a member of the chain.
+ * Must be a heap-only tuple unless it's the root item, and the root
+ * item happens to be a tuple with storage. The XMIN/XMAX cross-check
+ * would have failed otherwise, since priorXmax must have come from a
+ * HOT updated tuple.
+ */
+ Assert((nchain == 0 && !rootisredirect) ||
+ HeapTupleHeaderIsHeapOnly(htup));
+
+ /*
+ * OK, this tuple is indeed a member of the chain. Commit to this
+ * interpretation by marking the item as found while traversing a
+ * chain.
*/
chainitems[nchain++] = offnum;
+ prstate->fromvalidchain[offnum] = true;
/*
* Check tuple's visibility status.
*/
- tupdead = recent_dead = false;
+ tupdead = false;
switch (heap_prune_satisfies_vacuum(prstate, &tup, buffer))
{
@@ -663,7 +682,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
case HEAPTUPLE_RECENTLY_DEAD:
- recent_dead = true;
/*
* This tuple may soon become DEAD. Update the hint field so
@@ -679,6 +697,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* This tuple may soon become DEAD. Update the hint field so
* that the page is reconsidered for pruning in future.
*/
+ Assert(!tupdead);
heap_prune_record_prunable(prstate,
HeapTupleHeaderGetUpdateXid(htup));
break;
@@ -692,6 +711,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* But we don't. See related decisions about when to mark the
* page prunable in heapam.c.
*/
+ Assert(!tupdead);
break;
default:
@@ -700,11 +720,18 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * Remember the last DEAD tuple seen. We will advance past
- * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
- * but we can't advance past anything else. We have to make sure that
- * we don't miss any DEAD tuples, since DEAD tuples that still have
- * tuple storage after pruning will confuse VACUUM.
+ * Remember the last DEAD tuple seen from this HOT chain.
+ *
+ * We expect to sometimes find a RECENTLY_DEAD tuple after a DEAD
+ * tuple. When this happens, the RECENTLY_DEAD tuple will be treated
+ * as if it was DEAD all along. Manage that now.
+ *
+ * We're not just interested in DEAD and RECENTLY_DEAD tuples. We
+ * need to traverse each and every HOT chain exhaustively, in order to
+ * determine which heap-only tuples are part of a valid HOT chain.
+ * Heap-only tuples that cannot be located like this must not be part
+ * of a valid HOT chain. They are therefore processed during our
+ * second pass over the page.
*/
if (tupdead)
{
@@ -712,8 +739,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
}
- else if (!recent_dead)
- break;
/*
* If the tuple is not HOT-updated, then we are at the end of this
@@ -729,7 +754,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* Advance to next chain member.
*/
Assert(ItemPointerGetBlockNumber(&htup->t_ctid) ==
- BufferGetBlockNumber(buffer));
+ prstate->targetblkno);
offnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
}
@@ -741,6 +766,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
*/
if (OffsetNumberIsValid(latestdead))
{
+ int i;
+
/*
* Mark as unused each intermediate item that we are able to remove
* from the chain.
@@ -757,10 +784,13 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
/*
* If the root entry had been a normal tuple, we are deleting it, so
* count it in the result. But changing a redirect (even to DEAD
- * state) doesn't count.
+ * state) doesn't get counted in ndeleted.
*/
if (ItemIdIsNormal(rootlp))
+ {
+ Assert(!rootisredirect);
ndeleted++;
+ }
/*
* If the DEAD tuple is at the end of the chain, the entire chain is
@@ -787,6 +817,68 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
return ndeleted;
}
+/*
+ * Handle disconnected heap-only tuples during second pass over page. We
+ * always expect to mark these tuples as DEAD here.
+ *
+ * We need this to handle aborted HOT updates. We only do this for heap-only
+ * tuples that are not reachable by traversing any valid HOT chain. This
+ * happens when the parent tuple was re-updated (after earlier aborted
+ * update). The re-updating session is under no obligation to clean up the
+ * aborted tuple itself, and couldn't do so reliably if it wanted to (it would
+ * have to first acquire a super-exclusive lock).
+ *
+ * Pruning must never miss any DEAD tuples. DEAD tuples that still have tuple
+ * storage after pruning will confuse VACUUM.
+*/
+static int
+heap_prune_heaponly(Buffer buffer, OffsetNumber offnum, PruneState *prstate)
+{
+ Page dp = (Page) BufferGetPage(buffer);
+ ItemId lp;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ HTSV_Result res;
+
+ lp = PageGetItemId(dp, offnum);
+ Assert(ItemIdIsNormal(lp));
+ htup = (HeapTupleHeader) PageGetItem(dp, lp);
+
+ /*
+ * Caller must make sure that the tuple at 'offnum' is in fact a heap-only
+ * tuple that is disconnected from its HOT chain (i.e. isn't reachable
+ * through a HOT traversal that starts at any plausible-looking root item
+ * on the page).
+ */
+ Assert(!prstate->fromvalidchain[offnum]);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+
+ /*
+ * We expect that disconnected heap-only tuples must be from aborted
+ * transactions. Any RECENTLY_DEAD tuples we see here are really DEAD,
+ * but the heap_prune_satisfies_vacuum test is too coarse to detect it.
+ */
+ tup.t_len = ItemIdGetLength(lp);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
+ res = heap_prune_satisfies_vacuum(prstate, &tup, buffer);
+ if (res == HEAPTUPLE_DEAD || res == HEAPTUPLE_RECENTLY_DEAD)
+ {
+ heap_prune_record_unused(prstate, offnum);
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ }
+ else
+ Assert(false);
+
+ /*
+ * Should always be DEAD. A DEAD heap-only tuple is always counted in
+ * top-level ndeleted counter for pruning operation.
+ */
+ return 1;
+}
+
/* Record lowest soon-prunable XID */
static void
heap_prune_record_prunable(PruneState *prstate, TransactionId xid)
@@ -810,10 +902,6 @@ heap_prune_record_redirect(PruneState *prstate,
prstate->redirected[prstate->nredirected * 2] = offnum;
prstate->redirected[prstate->nredirected * 2 + 1] = rdoffnum;
prstate->nredirected++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
- Assert(!prstate->marked[rdoffnum]);
- prstate->marked[rdoffnum] = true;
}
/* Record line pointer to be marked dead */
@@ -823,8 +911,6 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
/* Record line pointer to be marked unused */
@@ -834,8 +920,6 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->nunused < MaxHeapTuplesPerPage);
prstate->nowunused[prstate->nunused] = offnum;
prstate->nunused++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-10 19:20 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 3 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-10 19:20 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-09 15:31:37 -0800, Peter Geoghegan wrote:
> I'm not sure why this seems to have become more of a problem following
> the snapshot scalability work from Andres -- Alexander mentioned that
> commit dc7420c2 looked like it was the source of the problem here, but
> I can't see any reason why that might be true (even though I accept
> that it might well *appear* to be true). I believe Andres has some
> theory on that, but I don't know the details myself. AFAICT, this is a
> live bug on all supported versions. We simply weren't being careful
> enough about breaking the invariant that an LP_REDIRECT can only point
> to a valid heap-only tuple. The really surprising thing here is that
> it took this long for it to visibly break.
The way this definitely breaks - I have been able to reproduce this in
isolation - is when one tuple is processed twice by heap_prune_chain(), and
the result of HeapTupleSatisfiesVacuum() changes from
HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
Consider a page like this:
lp 1: redirect to lp2
lp 2: deleted by xid x, not yet committed
and a sequence of events like this:
1) heap_prune_chain(rootlp = 1)
2) commit x
3) heap_prune_chain(rootlp = 2)
1) heap_prune_chain(rootlp = 1) will go to lp2, and see a
HEAPTUPLE_DELETE_IN_PROGRESS and thus not do anything.
3) then could, with the snapshot scalability changes, get DEAD back from
HTSV. Due to the "fuzzy" nature of the post-snapshot-scalability xid horizons,
that is possible, because we can end up rechecking the boundary condition and
seeing that now the horizon allows us to prune x / lp2.
At that point we have a redirect tuple pointing into an unused slot. Which is
"illegal", because something independent can be inserted into that slot.
What made this hard to understand (and likely hard to hit) is that we don't
recompute the xid horizons more than once per hot pruning ([1]). At first I
concluded that a change from RECENTLY_DEAD to DEAD could thus not happen - and
it doesn't: We go from HEAPTUPLE_DELETE_IN_PROGRESS to DEAD, which is possible
because there was no horizon test for HEAPTUPLE_DELETE_IN_PROGRESS.
Note that there are several paths < 14, that cause HTSV()'s answer to change
for the same xid. E.g. when the transaction inserting a tuple version aborts,
we go from HEAPTUPLE_INSERT_IN_PROGRESS to DEAD. But I haven't quite found a
path to trigger problems with that, because there won't be redirects to a
tuple version that is HEAPTUPLE_INSERT_IN_PROGRESS (but there can be redirects
to a HEAPTUPLE_DELETE_IN_PROGRESS or RECENTLY_DEAD).
I hit a crash once in 13 with a slightly evolved version of the test (many
connections creating / dropping the partitions as in the original scenario,
using :client_id to target different tables). It's possible that my
instrumentation was the cause of that. Unfortunately it took quite a few hours
to hit the problem in 13...
Greetings,
Andres Freund
[1] it's a bit more complicated than that, we only recompute the horizon when
a) we've not done it before in the current xact, b) RecentXmin changed during
a snapshot computation. Recomputing the horizon is expensive-ish, so we don't
want to do it constantly.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-10 21:04 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-10 21:04 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> The way this definitely breaks - I have been able to reproduce this in
> isolation - is when one tuple is processed twice by heap_prune_chain(), and
> the result of HeapTupleSatisfiesVacuum() changes from
> HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
I had no idea that that was now possible. I really think that this
ought to be documented centrally.
As you know, I don't like the way that vacuumlazy.c doesn't explain
anything about the relationship between OldestXmin (which still
exists, but isn't used for pruning), and the similar GlobalVisState
state (used only during pruning). Surely this deserves to be
addressed, because we expect these two things to agree in certain
specific ways. But not necessarily in others.
> Note that there are several paths < 14, that cause HTSV()'s answer to change
> for the same xid. E.g. when the transaction inserting a tuple version aborts,
> we go from HEAPTUPLE_INSERT_IN_PROGRESS to DEAD.
Right -- that one I certainly knew about. After all, the
tupgone-ectomy work from my commit 8523492d specifically targeted this
case.
> But I haven't quite found a
> path to trigger problems with that, because there won't be redirects to a
> tuple version that is HEAPTUPLE_INSERT_IN_PROGRESS (but there can be redirects
> to a HEAPTUPLE_DELETE_IN_PROGRESS or RECENTLY_DEAD).
That explains why the snapshot scalability either made these problems
possible for the first time, or at the very least made them far far
more likely in practice.
The relevant code in pruneheap.c was always incredibly fragile -- no
question. Even still, there is really no good reason to believe that
that was actually a problem before commit dc7420c2. Even if we assume
that there's a problem before 14, the surface area is vastly smaller
than on 14 -- the relevant pruneheap.c code hasn't really ever changed
since HOT went in. And so I think that the most sensible course of
action here is this: commit a fix to Postgres 14 + HEAD only -- no
backpatch to earlier versions.
We could go back further than that, but ISTM that the risk of causing
new problems far outweighs the benefits. Whereas I feel pretty
confident that we need to do something on 14.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-10 21:40 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-10 21:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 1:04 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> > The way this definitely breaks - I have been able to reproduce this in
> > isolation - is when one tuple is processed twice by heap_prune_chain(), and
> > the result of HeapTupleSatisfiesVacuum() changes from
> > HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
>
> I had no idea that that was now possible. I really think that this
> ought to be documented centrally.
BTW, is it just a coincidence that we have only seen these problems
with pg_class? Or does it have something to do with the specifics of
VISHORIZON_CATALOG/horizons.catalog_oldest_nonremovable, or something
else along those lines?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-10 22:18 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-10 22:18 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> I hit a crash once in 13 with a slightly evolved version of the test (many
> connections creating / dropping the partitions as in the original scenario,
> using :client_id to target different tables). It's possible that my
> instrumentation was the cause of that. Unfortunately it took quite a few hours
> to hit the problem in 13...
Have you thought about the case where a transaction does a HOT update
of the same row twice, and then aborts?
I'm asking because I notice that the fragile "We need this primarily
to handle aborted HOT updates" precheck for
HeapTupleHeaderIsHeapOnly() doesn't just check if the heap-only tuple
is DEAD before deciding to mark it LP_UNUSED. It also checks
HeapTupleHeaderIsHotUpdated() against the target tuple -- that's
another condition of the tuple being marked unused. Of course, whether
or not a given tuple is considered HeapTupleHeaderIsHotUpdated() can
change from true to false when an updater concurrently aborts. Could
that have race conditions?
In other words: what if the aforementioned "aborted HOT updates"
precheck code doesn't deal with a DEAD tuple, imagining that it's not
a relevant tuple, while at the same time the later HOT-chain-chasing
code *also* doesn't get to the tuple? What if they each assume that
the other will/has taken care of it, due to a race?
So far we've been worried about cases where these two code paths
clobber each other -- that's what we've actually seen. We should also
at least consider the possibility that we have the opposite problem,
which is what this really is.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 00:47 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-11 00:47 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 13:04:43 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> > The way this definitely breaks - I have been able to reproduce this in
> > isolation - is when one tuple is processed twice by heap_prune_chain(), and
> > the result of HeapTupleSatisfiesVacuum() changes from
> > HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
>
> I had no idea that that was now possible. I really think that this
> ought to be documented centrally.
Where would you suggest?
> The relevant code in pruneheap.c was always incredibly fragile -- no
> question. Even still, there is really no good reason to believe that
> that was actually a problem before commit dc7420c2. Even if we assume
> that there's a problem before 14, the surface area is vastly smaller
> than on 14 -- the relevant pruneheap.c code hasn't really ever changed
> since HOT went in. And so I think that the most sensible course of
> action here is this: commit a fix to Postgres 14 + HEAD only -- no
> backpatch to earlier versions.
Yea. The fact that I also saw *one* error in 13 worries me a bit, but perhaps
that was something else. Even if we eventually need to backpatch something
further, having it in 14/master first is good.
The fact that 13 didn't trigger the problem reliably doesn't necessarily much
- it's a pretty limited workload. There e.g. are no aborts.
I think we might be able to do something a bit more limited than what you
propose. But I'm not sure it's worth going for that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 00:57 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-11 00:57 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 14:18:01 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> > I hit a crash once in 13 with a slightly evolved version of the test (many
> > connections creating / dropping the partitions as in the original scenario,
> > using :client_id to target different tables). It's possible that my
> > instrumentation was the cause of that. Unfortunately it took quite a few hours
> > to hit the problem in 13...
>
> Have you thought about the case where a transaction does a HOT update
> of the same row twice, and then aborts?
Yes. I don't think it's problematic right now, because the redirect would, I
think, in all cases have to point to the chain element before those tuples,
because the preceding value would just have to be DELETE_IN_PROGRESS, which we
we don't follow in heap_prune_chain().
> I'm asking because I notice that the fragile "We need this primarily
> to handle aborted HOT updates" precheck for
> HeapTupleHeaderIsHeapOnly() doesn't just check if the heap-only tuple
> is DEAD before deciding to mark it LP_UNUSED. It also checks
> HeapTupleHeaderIsHotUpdated() against the target tuple -- that's
> another condition of the tuple being marked unused. Of course, whether
> or not a given tuple is considered HeapTupleHeaderIsHotUpdated() can
> change from true to false when an updater concurrently aborts. Could
> that have race conditions?
I wondered about that too, but I couldn't *quite* come up with a problematic
scenario, because I don't think any of the cases that can change
HeapTupleHeaderIsHotUpdated() would have allowed to set the redirect to a
subsequent chain element.
> In other words: what if the aforementioned "aborted HOT updates"
> precheck code doesn't deal with a DEAD tuple, imagining that it's not
> a relevant tuple, while at the same time the later HOT-chain-chasing
> code *also* doesn't get to the tuple? What if they each assume that
> the other will/has taken care of it, due to a race?
Then we'd just end up not pruning the tuple, I think. Which should be fine, as
it could only happen for fairly new tuples.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 01:03 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-11 01:03 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 13:40:25 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 1:04 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> > > The way this definitely breaks - I have been able to reproduce this in
> > > isolation - is when one tuple is processed twice by heap_prune_chain(), and
> > > the result of HeapTupleSatisfiesVacuum() changes from
> > > HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
> >
> > I had no idea that that was now possible. I really think that this
> > ought to be documented centrally.
>
> BTW, is it just a coincidence that we have only seen these problems
> with pg_class? Or does it have something to do with the specifics of
> VISHORIZON_CATALOG/horizons.catalog_oldest_nonremovable, or something
> else along those lines?
It's the testcase. The test instructions specify autovacuum=off and only
pg_class is vacuumed. There are a other hot updates than to pg_class, but
there's much less HOT pruning on them because there's no loop of VACUUMs.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 01:19 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 01:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 4:57 PM Andres Freund <andres@anarazel.de> wrote:
> Yes. I don't think it's problematic right now, because the redirect would, I
> think, in all cases have to point to the chain element before those tuples,
> because the preceding value would just have to be DELETE_IN_PROGRESS, which we
> we don't follow in heap_prune_chain().
Actually, I was more worried about versions before Postgres 14 here --
versions that don't have that new DELETE_IN_PROGRESS pruning behavior
you mentioned.
> > I'm asking because I notice that the fragile "We need this primarily
> > to handle aborted HOT updates" precheck for
> > HeapTupleHeaderIsHeapOnly() doesn't just check if the heap-only tuple
> > is DEAD before deciding to mark it LP_UNUSED. It also checks
> > HeapTupleHeaderIsHotUpdated() against the target tuple -- that's
> > another condition of the tuple being marked unused. Of course, whether
> > or not a given tuple is considered HeapTupleHeaderIsHotUpdated() can
> > change from true to false when an updater concurrently aborts. Could
> > that have race conditions?
>
> I wondered about that too, but I couldn't *quite* come up with a problematic
> scenario, because I don't think any of the cases that can change
> HeapTupleHeaderIsHotUpdated() would have allowed to set the redirect to a
> subsequent chain element.
It's pretty complicated.
> > In other words: what if the aforementioned "aborted HOT updates"
> > precheck code doesn't deal with a DEAD tuple, imagining that it's not
> > a relevant tuple, while at the same time the later HOT-chain-chasing
> > code *also* doesn't get to the tuple? What if they each assume that
> > the other will/has taken care of it, due to a race?
>
> Then we'd just end up not pruning the tuple, I think. Which should be fine, as
> it could only happen for fairly new tuples.
It's pretty far from ideal if the situation cannot correct itself in a
timely manner. Offhand I wonder if this might have something to do
with remaining suspected cases where we restart pruning during VACUUM,
based on the belief that an inserter concurrently aborted. If the DEAD
tuple is unreachable by pruning *forever*, then we're in trouble.
The good news is that the fix for missing DEAD tuples (a hypothetical
problem) is the same as the fix for the known, concrete problem:
process whole HOT chains first, and only later (in a "second pass"
over the page) process remaining heap-only tuples -- those heap-only
tuples that cannot be located any other way (because we tried and
failed).
You probably noticed that my patch does *not* have the same
"!HeapTupleHeaderIsHotUpdated()" check for these
unreachable-via-HOT-chain heap-only tuples. The mere fact that they're
unreachable is a good enough reason to consider them DEAD (and so mark
them LP_UNUSED). We do check heap_prune_satisfies_vacuum() for these
tuples too, but that is theoretically unnecessary. This feels pretty
water tight to me -- contradictory interpretations of what heap-only
tuple is and is in what HOT chain (if any) now seem impossible.
I benchmarked the patch, and it looks like there is a consistent
reduction in latency -- which you probably won't find too surprising.
That isn't the goal, but it is nice to see that new performance
regressions are unlikely to be a problem for the bug fix.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 01:37 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 01:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 4:47 PM Andres Freund <andres@anarazel.de> wrote:
> On 2021-11-10 13:04:43 -0800, Peter Geoghegan wrote:
> > On Wed, Nov 10, 2021 at 11:20 AM Andres Freund <andres@anarazel.de> wrote:
> > > The way this definitely breaks - I have been able to reproduce this in
> > > isolation - is when one tuple is processed twice by heap_prune_chain(), and
> > > the result of HeapTupleSatisfiesVacuum() changes from
> > > HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
> >
> > I had no idea that that was now possible. I really think that this
> > ought to be documented centrally.
>
> Where would you suggest?
Offhand I'd say that it would be a good idea to add comments over the
call to vacuum_set_xid_limits() made from vacuumlazy.c.
You might also move the call to GlobalVisTestFor() out of
lazy_scan_heap(), so that it gets called right after
vacuum_set_xid_limits(). That would make the new explanation easier to
follow, since you are after all explaining the relationship between
OldestXmin (or the vacuum_set_xid_limits() call itself) and vistest
(or the GlobalVisTestFor() call itself).
Why do they have to be called in that order? Or do they? I noticed
that "make check-world" won't break if you switch the order.
I assume that you're going to want to say something about what needs
to happen in lazy_scan_prune() in these new comments -- since that is
where the relationship between these two things is most crucial.
Thanks
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 01:43 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-11 01:43 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 17:19:14 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 4:57 PM Andres Freund <andres@anarazel.de> wrote:
> > Yes. I don't think it's problematic right now, because the redirect would, I
> > think, in all cases have to point to the chain element before those tuples,
> > because the preceding value would just have to be DELETE_IN_PROGRESS, which we
> > we don't follow in heap_prune_chain().
>
> Actually, I was more worried about versions before Postgres 14 here --
> versions that don't have that new DELETE_IN_PROGRESS pruning behavior
> you mentioned.
I think we're actually saying the same thing. I.e. that we, so far, don't have
a concrete reason to worry about pre 14 versions.
> > > In other words: what if the aforementioned "aborted HOT updates"
> > > precheck code doesn't deal with a DEAD tuple, imagining that it's not
> > > a relevant tuple, while at the same time the later HOT-chain-chasing
> > > code *also* doesn't get to the tuple? What if they each assume that
> > > the other will/has taken care of it, due to a race?
> >
> > Then we'd just end up not pruning the tuple, I think. Which should be fine, as
> > it could only happen for fairly new tuples.
>
> It's pretty far from ideal if the situation cannot correct itself in a
> timely manner.
Wouldn't it immediately corrected when called from vacuum due to the DEAD
check? Afaict this is a one-way ratched, so the next prune is guaranteed to
get it?
> Offhand I wonder if this might have something to do with remaining suspected
> cases where we restart pruning during VACUUM, based on the belief that an
> inserter concurrently aborted. If the DEAD tuple is unreachable by pruning
> *forever*, then we're in trouble.
But why would it be unreachable forever? The next hot prune will see
!HeapTupleHeaderIsHotUpdated() and process it via the "If the tuple is DEAD
and doesn't chain to anything else" block?
> You probably noticed that my patch does *not* have the same
> "!HeapTupleHeaderIsHotUpdated()" check for these
> unreachable-via-HOT-chain heap-only tuples. The mere fact that they're
> unreachable is a good enough reason to consider them DEAD (and so mark
> them LP_UNUSED). We do check heap_prune_satisfies_vacuum() for these
> tuples too, but that is theoretically unnecessary. This feels pretty
> water tight to me -- contradictory interpretations of what heap-only
> tuple is and is in what HOT chain (if any) now seem impossible.
Hm. I guess you *have* to actually process them regardless of
!HeapTupleHeaderIsHotUpdated() with the new approach, because otherwise we'd
potentially only process only the tail item of a disconnected chain in each
heap_hot_prune() call? Although that might be unreachable due to the HTSV
calls likely breaking the chain in all disconnected cases (but perhaps not
with multixacts, there could be remaining lockers).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 01:59 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 01:59 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 5:43 PM Andres Freund <andres@anarazel.de> wrote:
> > Actually, I was more worried about versions before Postgres 14 here --
> > versions that don't have that new DELETE_IN_PROGRESS pruning behavior
> > you mentioned.
>
> I think we're actually saying the same thing. I.e. that we, so far, don't have
> a concrete reason to worry about pre 14 versions.
Mostly, yes. I think that it's not a good idea to backpatch a fix
beyond 14 at this time.
All I was saying beyond that was: If there was a problem before 14,
then it would probably be a problem with the
"HEAPTUPLE_INSERT_IN_PROGRESS becomes DEAD" case. And, it's probably
also more likely that it would be "leaking DEAD tuples", since that is
much less obvious than the bug we know about, in 14. In general
problems with "incorrectly broken" HOT chains are very subtle, and
less well understood than other problems.
> Wouldn't it immediately corrected when called from vacuum due to the DEAD
> check? Afaict this is a one-way ratched, so the next prune is guaranteed to
> get it?
I believe that's true, but I still worry just a little about extreme
cases. I'm not concerned enough to advocate backpatch beyond 14 (not
even close!), at least for now. And so my concern is pretty abstract
and theoretical at this point.
It'll be nice to obviously not have the theoretical problem on 14,
which needs a fix for the non-theoretical problem anyway -- AFAICT
there is no extra risk anyway.
> But why would it be unreachable forever? The next hot prune will see
> !HeapTupleHeaderIsHotUpdated() and process it via the "If the tuple is DEAD
> and doesn't chain to anything else" block?
I agree.
> > You probably noticed that my patch does *not* have the same
> > "!HeapTupleHeaderIsHotUpdated()" check for these
> > unreachable-via-HOT-chain heap-only tuples. The mere fact that they're
> > unreachable is a good enough reason to consider them DEAD (and so mark
> > them LP_UNUSED). We do check heap_prune_satisfies_vacuum() for these
> > tuples too, but that is theoretically unnecessary. This feels pretty
> > water tight to me -- contradictory interpretations of what heap-only
> > tuple is and is in what HOT chain (if any) now seem impossible.
>
> Hm. I guess you *have* to actually process them regardless of
> !HeapTupleHeaderIsHotUpdated() with the new approach, because otherwise we'd
> potentially only process only the tail item of a disconnected chain in each
> heap_hot_prune() call?
That's true.
> Although that might be unreachable due to the HTSV
> calls likely breaking the chain in all disconnected cases (but perhaps not
> with multixacts, there could be remaining lockers).
Also true. But I like that we don't really have to worry about it with
the fix in place.
If we cannot reach a heap-only tuple via a valid HOT chain, it's
already effectively DEAD, anyway -- all bets are off. We could even
add a "can't happen" ERROR to catch the case where HTSV disagrees.
Unsure if that would be worth it, but it's certainly worth an
assertion -- which I've already added.
What do you think of the idea of a "can't happen" ERROR here?
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 02:16 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-11 02:16 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 17:37:38 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 4:47 PM Andres Freund <andres@anarazel.de> wrote:
> Offhand I'd say that it would be a good idea to add comments over the
> call to vacuum_set_xid_limits() made from vacuumlazy.c.
Hm. To me all of this is more general than vacuum[lazy].c. Or even than
anything heap related.
> You might also move the call to GlobalVisTestFor() out of
> lazy_scan_heap(), so that it gets called right after
> vacuum_set_xid_limits(). That would make the new explanation easier to
> follow, since you are after all explaining the relationship between
> OldestXmin (or the vacuum_set_xid_limits() call itself) and vistest
> (or the GlobalVisTestFor() call itself).
>
> Why do they have to be called in that order? Or do they? I noticed
> that "make check-world" won't break if you switch the order.
We need pruning to be at least as aggressive as relfrozenxid. If we did it the
other way round, we couldn't guarantee that.
I think we should work towards not actually using a statically determined
relfrozenxid. We cause a lot of unnecessary re-vacuuming by using a static
cutoff - instead we should check what the actually oldest xid in the table is
and set relfrozenxid to that.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 02:20 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-11 02:20 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 18:16:18 -0800, Andres Freund wrote:
> On 2021-11-10 17:37:38 -0800, Peter Geoghegan wrote:
> > You might also move the call to GlobalVisTestFor() out of
> > lazy_scan_heap(), so that it gets called right after
> > vacuum_set_xid_limits(). That would make the new explanation easier to
> > follow, since you are after all explaining the relationship between
> > OldestXmin (or the vacuum_set_xid_limits() call itself) and vistest
> > (or the GlobalVisTestFor() call itself).
> >
> > Why do they have to be called in that order? Or do they? I noticed
> > that "make check-world" won't break if you switch the order.
>
> We need pruning to be at least as aggressive as relfrozenxid. If we did it the
> other way round, we couldn't guarantee that.
There's a relevant comment above ComputeXidHorizons():
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
* heap vacuum's call to heap_page_prune() uses a more conservative horizon
* than later when deciding which tuples can be removed - which the code
* doesn't expect (breaking HOT).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 02:40 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 02:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 6:16 PM Andres Freund <andres@anarazel.de> wrote:
> Hm. To me all of this is more general than vacuum[lazy].c. Or even than
> anything heap related.
Here is a sensible compromise: put most of what you want to say
wherever (I guess procarray.c), and then move the vacuumlazy.c call to
GlobalVisTestFor() back, so that it comes immediately after the
vacuum_set_xid_limits() call. Then place a few breadcrumb comments
that reference the place in procarray.c that has the real discussion.
> We need pruning to be at least as aggressive as relfrozenxid. If we did it the
> other way round, we couldn't guarantee that.
I thought that that's what it was, but the code doesn't actually say
anything about it. The distance between the two actually-related
things is jarring, at least to me.
> I think we should work towards not actually using a statically determined
> relfrozenxid. We cause a lot of unnecessary re-vacuuming by using a static
> cutoff - instead we should check what the actually oldest xid in the table is
> and set relfrozenxid to that.
I agree, but that doesn't seem relevant to me. AFAICT the "effective"
relfrozenxid when applying this hypothetical future optimization (the
"actually oldest xid in the table", as you put it) must never end up
exceeding the original OldestXmin cutoff. And so I don't think that it
changes the fundamental invariants for either OldestXmin, or for
freezeLimit/relfrozenxid. Specifically, the "freezeLimit <=
OldestXmin" invariant.
We could probably *also* freeze tuples opportunistically (e.g., freeze
a few tuples on a page early to be able to mark it all-frozen sooner),
since freezing is basically just an all-visible marking that applies
at the tuple level. We could perhaps even do this when the tuples
would not be visible to our original OldestXmin (they just have to be
visible to every possible MVCC snapshot, and so VACUUM's OldestXmin
itself doesn't necessarily have to be considered). This additional
optimization doesn't seem like it changes the invariants, either,
though. Since I'm pretty sure that freezing tuples early isn't
compatible with allowing those tuples to affect the final
freezeLimit/relfrozenxid (when we have both optimization, working
together).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 03:08 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-11 03:08 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-09 19:07:02 -0800, Peter Geoghegan wrote:
> diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
> index c7331d810..31fa4d2a3 100644
> --- a/src/backend/access/heap/pruneheap.c
> +++ b/src/backend/access/heap/pruneheap.c
> @@ -31,6 +31,7 @@
> typedef struct
> {
> Relation rel;
> + BlockNumber targetblkno;
It's a good optimization to remove the reundant BufferGetBlockNumber(), but
perhaps it's worth committing that separately?
> @@ -256,10 +260,8 @@ heap_page_prune(Relation relation, Buffer buffer,
> offnum <= maxoff;
> offnum = OffsetNumberNext(offnum))
> {
> - ItemId itemid;
> -
> /* Ignore items already processed as part of an earlier chain */
> - if (prstate.marked[offnum])
> + if (prstate.fromvalidchain[offnum])
> continue;
>
> /*
ISTM that we should now call heap_prune_chain() only for the start of actual
chains (i.e. redirect or non-hot tuple).
I think it'd be good to have a comment above the loop explaining that we're
just iterating over chains starting at a non-HOT element.
> @@ -269,15 +271,29 @@ heap_page_prune(Relation relation, Buffer buffer,
> if (off_loc)
> *off_loc = offnum;
>
> - /* Nothing to do if slot is empty or already dead */
> - itemid = PageGetItemId(page, offnum);
> - if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
> - continue;
> -
> /* Process this item or chain of items */
> ndeleted += heap_prune_chain(buffer, offnum, &prstate);
> }
Why did you move this into heap_prune_chain()?
> + /*
> + * Scan the page again, processing any heap-only tuples that we now know
> + * are not part of any valid HOT chain
> + */
> + for (offnum = FirstOffsetNumber;
> + offnum <= maxoff;
> + offnum = OffsetNumberNext(offnum))
> + {
> + /* Ignore items already processed as part of a known valid chain */
> + if (prstate.fromvalidchain[offnum])
> + continue;
> +
> + if (off_loc)
> + *off_loc = offnum;
> +
> + /* Process this disconnected heap-only tuple */
> + ndeleted += heap_prune_heaponly(buffer, offnum, &prstate);
> + }
> +
> /* Clear the offset information once we have processed the given page. */
> if (off_loc)
> *off_loc = InvalidOffsetNumber;
> @@ -383,19 +399,8 @@ heap_page_prune(Relation relation, Buffer buffer,
> pgstat_update_heap_dead_tuples(relation, ndeleted - prstate.ndead);
>
> /*
> - * XXX Should we update the FSM information of this page ?
> - *
> - * There are two schools of thought here. We may not want to update FSM
> - * information so that the page is not used for unrelated UPDATEs/INSERTs
> - * and any free space in this page will remain available for further
> - * UPDATEs in *this* page, thus improving chances for doing HOT updates.
> - *
> - * But for a large table and where a page does not receive further UPDATEs
> - * for a long time, we might waste this space by not updating the FSM
> - * information. The relation may get extended and fragmented further.
> - *
> - * One possibility is to leave "fillfactor" worth of space in this page
> - * and update FSM with the remaining space.
> + * Don't update the FSM information on this page. We leave it up to our
> + * caller to decide what to do about that.
> */
I don't think this should be part of the commit?
> offnum = rootoffnum;
> + nchain = 0;
> /* while not end of the chain */
> for (;;)
> {
> ItemId lp;
> - bool tupdead,
> - recent_dead;
> + HeapTupleHeader htup;
> + HeapTupleData tup;
> + bool tupdead;
>
> /* Sanity check (pure paranoia) */
> if (offnum < FirstOffsetNumber)
> @@ -601,15 +603,11 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> break;
>
> /* If item is already processed, stop --- it must not be same chain */
> - if (prstate->marked[offnum])
> + if (nchain != 0 && prstate->fromvalidchain[offnum])
> break;
I think we should make it an error to reach the same tuple multiple ways. It's
not *quite* as trivial as making this an assert/error though, as we can only
raise an error after checking that the xmin/xmax thing passes.
> /*
> * If we are looking at the redirected root line pointer, jump to the
> * first normal tuple in the chain. If we find a redirect somewhere
> @@ -619,42 +617,63 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> {
> if (nchain > 0)
> break; /* not at start of chain */
> + Assert(rootisredirect);
> chainitems[nchain++] = offnum;
> offnum = ItemIdGetRedirect(rootlp);
> continue;
> }
Why are we handling this inside the loop?
> - /*
> - * Likewise, a dead line pointer can't be part of the chain. (We
> - * already eliminated the case of dead root tuple outside this
> - * function.)
> - */
> - if (ItemIdIsDead(lp))
> + /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
> + if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
> + {
> + /*
> + * XXX What if we just came from root item, which is a plain heap
> + * tuple? Do we need assert that notices when we reach an LP_DEAD
> + * or LP_UNUSED item having started from such a root item?
> + */
> + Assert(!rootisredirect || nchain > 1);
I don't think we can assert that. It's perfectly possible to have a non-hot
update chain where the following element can be vacuumed, but the preceding
element can't (e.g. when the updater has an older xid that prevents it from
being pruned).
> chainitems[nchain++] = offnum;
> + prstate->fromvalidchain[offnum] = true;
>
> /*
> * Check tuple's visibility status.
> */
> - tupdead = recent_dead = false;
> + tupdead = false;
>
> switch (heap_prune_satisfies_vacuum(prstate, &tup, buffer))
> {
> @@ -663,7 +682,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> break;
>
> case HEAPTUPLE_RECENTLY_DEAD:
> - recent_dead = true;
>
> /*
> * This tuple may soon become DEAD. Update the hint field so
> @@ -679,6 +697,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> * This tuple may soon become DEAD. Update the hint field so
> * that the page is reconsidered for pruning in future.
> */
> + Assert(!tupdead);
> heap_prune_record_prunable(prstate,
> HeapTupleHeaderGetUpdateXid(htup));
> break;
> @@ -692,6 +711,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> * But we don't. See related decisions about when to mark the
> * page prunable in heapam.c.
> */
> + Assert(!tupdead);
> break;
>
> default:
I don't understand these new assert? We just set tupdead to false above?
> @@ -700,11 +720,18 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> }
>
> /*
> - * Remember the last DEAD tuple seen. We will advance past
> - * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
> - * but we can't advance past anything else. We have to make sure that
> - * we don't miss any DEAD tuples, since DEAD tuples that still have
> - * tuple storage after pruning will confuse VACUUM.
> + * Remember the last DEAD tuple seen from this HOT chain.
> + *
> + * We expect to sometimes find a RECENTLY_DEAD tuple after a DEAD
> + * tuple. When this happens, the RECENTLY_DEAD tuple will be treated
> + * as if it was DEAD all along. Manage that now.
I now actually wonder why this is correct. There's another comment about it:
> We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
> * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
But that doesn't justify very much.
What prevents the scenario that some other backend e.g. has a snapshot with
xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
row, but *not* the RECENTLY_DEAD one.
> + * We're not just interested in DEAD and RECENTLY_DEAD tuples. We
> + * need to traverse each and every HOT chain exhaustively, in order to
> + * determine which heap-only tuples are part of a valid HOT chain.
> + * Heap-only tuples that cannot be located like this must not be part
> + * of a valid HOT chain. They are therefore processed during our
> + * second pass over the page.
> */
> if (tupdead)
> {
This comment can't really be understood without the historical behaviour of
the function.
> +static int
> +heap_prune_heaponly(Buffer buffer, OffsetNumber offnum, PruneState *prstate)
> +{
> + Page dp = (Page) BufferGetPage(buffer);
> + ItemId lp;
> + HeapTupleHeader htup;
> + HeapTupleData tup;
> + HTSV_Result res;
> +
> + lp = PageGetItemId(dp, offnum);
> + Assert(ItemIdIsNormal(lp));
> + htup = (HeapTupleHeader) PageGetItem(dp, lp);
> +
> + /*
> + * Caller must make sure that the tuple at 'offnum' is in fact a heap-only
> + * tuple that is disconnected from its HOT chain (i.e. isn't reachable
> + * through a HOT traversal that starts at any plausible-looking root item
> + * on the page).
> + */
> + Assert(!prstate->fromvalidchain[offnum]);
> + Assert(HeapTupleHeaderIsHeapOnly(htup));
> +
> + /*
> + * We expect that disconnected heap-only tuples must be from aborted
> + * transactions. Any RECENTLY_DEAD tuples we see here are really DEAD,
> + * but the heap_prune_satisfies_vacuum test is too coarse to detect it.
> + */
> + tup.t_len = ItemIdGetLength(lp);
> + tup.t_tableOid = RelationGetRelid(prstate->rel);
> + tup.t_data = htup;
> + ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
> + res = heap_prune_satisfies_vacuum(prstate, &tup, buffer);
> + if (res == HEAPTUPLE_DEAD || res == HEAPTUPLE_RECENTLY_DEAD)
> + {
> + heap_prune_record_unused(prstate, offnum);
> + HeapTupleHeaderAdvanceLatestRemovedXid(htup,
> + &prstate->latestRemovedXid);
> + }
> + else
> + Assert(false);
> +
> + /*
> + * Should always be DEAD. A DEAD heap-only tuple is always counted in
> + * top-level ndeleted counter for pruning operation.
> + */
> + return 1;
> +}
It seems weird to have the if (res == HEAPTUPLE_DEAD ..) branch, but then to
unconditionally return 1.
I'm not actually sure the Assert is unreachable. I can imagine cases where
we'd see e.g. DELETE/INSERT_IN_PROGRESS due to a concurrent subtransaction
abort or such.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 03:16 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-11 03:16 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 18:40:13 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 6:16 PM Andres Freund <andres@anarazel.de> wrote:
> > Hm. To me all of this is more general than vacuum[lazy].c. Or even than
> > anything heap related.
>
> Here is a sensible compromise: put most of what you want to say
> wherever (I guess procarray.c), and then move the vacuumlazy.c call to
> GlobalVisTestFor() back, so that it comes immediately after the
> vacuum_set_xid_limits() call. Then place a few breadcrumb comments
> that reference the place in procarray.c that has the real discussion.
I don't really think it's a good idea to move it to earlier. If anything I
would want to move it later (and eventually occasionally re-compute it, that'd
be a *huge* win for longrunning vacuums). We can't really do the same with the
hard horizons. It's currently super likely that the horizon moves forward a
lot between vacuum_set_xid_limits() and GlobalVisTestFor(), but moving it
closer still is going in the wrong direction.
So I'm inclined to add a comment saying that we need to do the
GlobalVisTestFor(), pointing to the comment in ComputeXidHorizons(), and
adding a TODO that we should find a heuristic when to recompute horizons?
> We could probably *also* freeze tuples opportunistically (e.g., freeze
> a few tuples on a page early to be able to mark it all-frozen sooner),
We should definitely do that when a page is already being dirtied for other
reasons.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 03:33 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 03:33 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 7:16 PM Andres Freund <andres@anarazel.de> wrote:
> I don't really think it's a good idea to move it to earlier. If anything I
> would want to move it later (and eventually occasionally re-compute it, that'd
> be a *huge* win for longrunning vacuums).
I agree that it would be a huge win, especially when we want to set
the VM after heap vacuuming, during the second heap pass. But...
> We can't really do the same with the
> hard horizons. It's currently super likely that the horizon moves forward a
> lot between vacuum_set_xid_limits() and GlobalVisTestFor(), but moving it
> closer still is going in the wrong direction.
...why is that super likely? I have to admit that I have no idea what
you mean by that.
> So I'm inclined to add a comment saying that we need to do the
> GlobalVisTestFor(), pointing to the comment in ComputeXidHorizons(), and
> adding a TODO that we should find a heuristic when to recompute horizons?
Okay. Even still, I suggest that you say something about this new
DELETE_IN_PROGRESS pruning behavior in vacuumlazy.c. Maybe in
lazy_scan_prune()?
> > We could probably *also* freeze tuples opportunistically (e.g., freeze
> > a few tuples on a page early to be able to mark it all-frozen sooner),
>
> We should definitely do that when a page is already being dirtied for other
> reasons.
Agreed. But at the very least we should be thinking about the
whole-page picture. If we made freezing a whole page cheap enough
(smaller WAL records), then maybe the separate all-visible state
becomes totally unnecessary (except for pg_upgrade).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 20:05 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-11 20:05 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 19:33:01 -0800, Peter Geoghegan wrote:
> On Wed, Nov 10, 2021 at 7:16 PM Andres Freund <andres@anarazel.de> wrote:
> > We can't really do the same with the
> > hard horizons. It's currently super likely that the horizon moves forward a
> > lot between vacuum_set_xid_limits() and GlobalVisTestFor(), but moving it
> > closer still is going in the wrong direction.
>
> ...why is that super likely? I have to admit that I have no idea what
> you mean by that.
Err, I missed a *not* in there, sorry for the confusion.
> > So I'm inclined to add a comment saying that we need to do the
> > GlobalVisTestFor(), pointing to the comment in ComputeXidHorizons(), and
> > adding a TODO that we should find a heuristic when to recompute horizons?
>
> Okay. Even still, I suggest that you say something about this new
> DELETE_IN_PROGRESS pruning behavior in vacuumlazy.c. Maybe in
> lazy_scan_prune()?
That doesn't quite make sense to me. There's quite a few ways that a HTSV can
change for the same tuple, and listing them all in vacuumlazy.c doesn't really
help people notice that when they add a new call or such. There's plenty
other callsites - like the one in heap_prune_chain()...
How about changing the comment above lazy_scan_prune() to note that the
concurrent abort case is one example of this behaviour? And then add a comment
to HeapTupleSatisfiesVacuum() along the lines of:
NB: Repeated calls to HeapTupleSatisfiesVacuum for the same tuple may change
even while the buffer is locked. Cases of this include:
- the transaction that inserted a tuple (version) aborted, changing the result
from HEAPTUPLE_INSERT_IN_PROGRESS to HEAPTUPLE_DEAD
- when used with GlobalVisTestIsRemovableXid(), HEAPTUPLE_DELETE_IN_PROGRESS
can change to HEAPTUPLE_DEAD if the horizon is updated
However, HEAPTUPLE_DEAD will not change back to HEAPTUPLE_RECENTLY_DEAD or
such.
> > > We could probably *also* freeze tuples opportunistically (e.g., freeze
> > > a few tuples on a page early to be able to mark it all-frozen sooner),
> >
> > We should definitely do that when a page is already being dirtied for other
> > reasons.
>
> Agreed. But at the very least we should be thinking about the
> whole-page picture. If we made freezing a whole page cheap enough
> (smaller WAL records), then maybe the separate all-visible state
> becomes totally unnecessary (except for pg_upgrade).
I doubt it. But I think getting rid of unnecessarily dirtying the page a
second (or third) time would be a big enough win already...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 20:25 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 20:25 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Nov 11, 2021 at 12:05 PM Andres Freund <andres@anarazel.de> wrote:
> > ...why is that super likely? I have to admit that I have no idea what
> > you mean by that.
>
> Err, I missed a *not* in there, sorry for the confusion.
Can you at least move the GlobalVisTestFor() call back? Move it from
lazy_scan_heap() to the point right before we call lazy_scan_heap(),
from heap_vacuum_rel()? That still seems like a big improvement, while
obviously having no real side-effects.
> > Okay. Even still, I suggest that you say something about this new
> > DELETE_IN_PROGRESS pruning behavior in vacuumlazy.c. Maybe in
> > lazy_scan_prune()?
>
> That doesn't quite make sense to me. There's quite a few ways that a HTSV can
> change for the same tuple, and listing them all in vacuumlazy.c doesn't really
> help people notice that when they add a new call or such. There's plenty
> other callsites - like the one in heap_prune_chain()...
Well, the retry in lazy_scan_prune() says that it only expects to have
to retry in the event of an aborted transaction. If nothing else, it
seems like you might want to amend that. If in fact the same
"DELETE_IN_PROGRESS becomes DEAD" issue applies (can't see why it
wouldn't).
> How about changing the comment above lazy_scan_prune() to note that the
> concurrent abort case is one example of this behaviour? And then add a comment
> to HeapTupleSatisfiesVacuum() along the lines of:
>
> NB: Repeated calls to HeapTupleSatisfiesVacuum for the same tuple may change
> even while the buffer is locked. Cases of this include:
> - the transaction that inserted a tuple (version) aborted, changing the result
> from HEAPTUPLE_INSERT_IN_PROGRESS to HEAPTUPLE_DEAD
> - when used with GlobalVisTestIsRemovableXid(), HEAPTUPLE_DELETE_IN_PROGRESS
> can change to HEAPTUPLE_DEAD if the horizon is updated
>
> However, HEAPTUPLE_DEAD will not change back to HEAPTUPLE_RECENTLY_DEAD or
> such.
Okay - sounds good.
> > Agreed. But at the very least we should be thinking about the
> > whole-page picture. If we made freezing a whole page cheap enough
> > (smaller WAL records), then maybe the separate all-visible state
> > becomes totally unnecessary (except for pg_upgrade).
>
> I doubt it. But I think getting rid of unnecessarily dirtying the page a
> second (or third) time would be a big enough win already...
Why do you doubt it? Do you suspect that the additional cost of
freezing the whole page (relative to just setting it all-visible)
cannot be reduced to the point where it can be ignored? That might be
true, but I'd like to know if that's what you meant. Because I can't
think of any other reason, and it's not like I haven't thought about
it a fair bit already.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-11 20:40 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-11 20:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Nov 11, 2021 at 12:25 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Can you at least move the GlobalVisTestFor() call back? Move it from
> lazy_scan_heap() to the point right before we call lazy_scan_heap(),
> from heap_vacuum_rel()? That still seems like a big improvement, while
> obviously having no real side-effects.
I imagine that you'd also move vistest into the main state struct
(vacrel) -- it could go right next to the existing OldestXmin field in
vacrel/LVRelState. These two things are too similar to not put next to
each other, and address together. We never use one without using the
other (except in heap_page_is_all_visible(), but that's just a
stripped-down version of lazy_scan_prune() anyway, so it shouldn't
count).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 00:58 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 00:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Nov 10, 2021 at 7:08 PM Andres Freund <andres@anarazel.de> wrote:
> It's a good optimization to remove the reundant BufferGetBlockNumber(), but
> perhaps it's worth committing that separately?
Ordinarily I would, but the intention here is to avoid new performance
regressions. Is it okay if I backpatch this performance optimization,
trivial though it may be? Or should I just do that on HEAD?
Attached is v3, which works though some of your feedback, though there
are still unresolved issues. I expect that we'll need to very
carefully review this bugfix, since it's a particularly tricky issue.
I expect to produce several more revisions.
> > @@ -256,10 +260,8 @@ heap_page_prune(Relation relation, Buffer buffer,
> > offnum <= maxoff;
> > offnum = OffsetNumberNext(offnum))
> > {
> > - ItemId itemid;
> > -
> > /* Ignore items already processed as part of an earlier chain */
> > - if (prstate.marked[offnum])
> > + if (prstate.fromvalidchain[offnum])
> > continue;
> >
> > /*
>
> ISTM that we should now call heap_prune_chain() only for the start of actual
> chains (i.e. redirect or non-hot tuple).
Seems easier to put the "heap-only tuple at rootoffnum, so it might be
disconnected -- not for me to process unless and until I find it
during later call that uses this item's HOT chain's root offset
number" logic here. That's at least closer to what we have already.
Let me know if you still don't like it in v3, though. I might just
change it, but it's related to the way that the array is now called
"visited" -- a question that I'll consider in the paragraph
immediately following this one.
> I think it'd be good to have a comment above the loop explaining that we're
> just iterating over chains starting at a non-HOT element.
You mean like heap_get_root_tuples() does it, with its own root_offsets array?
I have changed the array -- it's now "visited". It's no longer
specifically about HOT chains. It's actually more like "definitely not
disconnected heap-only tuple" items, but I didn't call it that for the
obvious reason. By the end of the first pass over the heap, all items
that are not heap-only tuples (as well as most heap-only tuples) will
be marked visited in v3.
The advantage of not explicitly making it about root-ness is that it
avoids relying on the already-squishy definition of "root item". We
want to visit an LP_DEAD or LP_UNUSED item in the first pass over the
page -- we must get them out of the way during the second pass (we
don't expect to have to deal with any tuple that isn't a disconnected
heap-only tuple at that point).
Another advantage of the "visited" array design is that it doesn't
matter if we happen to visit an LP_DEAD/LP_UNUSED item via a heap-only
tuples bogus-but-valid t_ctid chain. Might as well just mark it
"visited' at that point, avoiding a second visit during the first heap
pass (in addition to the second pass). That micro optimization may not
be worth much, but it feels kind of natural to me. What do you think?
> > @@ -269,15 +271,29 @@ heap_page_prune(Relation relation, Buffer buffer,
> > if (off_loc)
> > *off_loc = offnum;
> >
> > - /* Nothing to do if slot is empty or already dead */
> > - itemid = PageGetItemId(page, offnum);
> > - if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
> > - continue;
> > -
> > /* Process this item or chain of items */
> > ndeleted += heap_prune_chain(buffer, offnum, &prstate);
> > }
>
> Why did you move this into heap_prune_chain()?
I find it simpler that way. Now both the first and second pass loops
have essentially the same structure as each other.
> >
> > /*
> > - * XXX Should we update the FSM information of this page ?
> > - *
> I don't think this should be part of the commit?
Agreed. Pushed that as a separate commit just now.
> I think we should make it an error to reach the same tuple multiple ways. It's
> not *quite* as trivial as making this an assert/error though, as we can only
> raise an error after checking that the xmin/xmax thing passes.
But as you go on to say, we should *not* do that for LP_DEAD or
LP_UNUSED items (nor heap-only tuples that don't pass xmin/xmax
crosscheck) -- since in general a heap-only tuples's t_ctid link is
allowed to point to just about anything (e.g., past the end of a heap
page's line pointer array).
Wouldn't it be simpler and more robust if we leaned on
heap_prune_satisfies_vacuum() for this instead? There will end up
being some "disconnected" remaining heap-only tuples when the same
tuple is reachable multiple ways. It's very likely that these
apparently-disconnected-but-who-knows tuples will fail some kind of
validation by heap_prune_satisfies_vacuum() in the second pass over
the page -- we expect to be able to treat them DEAD, which provides a
good cross-check. (Granted, there is an unresolved question about what
exact HTSV return codes we can expect here, but I imagine that a LIVE
disconnected tuple will be fully impossible, for example).
Note also that v3 directly Asserts() for the truly important case: The
case where a HOT chain whose root item is an LP_REDIRECT has only one
valid-looking item (the root item itself). That assertion is near the
end of heap_prune_chain() in v3. This assertion more or less replaces
the very dangerous "We found a redirect item that doesn't point to a
valid follow-on" block from HEAD.
> > /*
> > * If we are looking at the redirected root line pointer, jump to the
> > * first normal tuple in the chain. If we find a redirect somewhere
> > @@ -619,42 +617,63 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> > {
> > if (nchain > 0)
> > break; /* not at start of chain */
> > + Assert(rootisredirect);
> > chainitems[nchain++] = offnum;
> > offnum = ItemIdGetRedirect(rootlp);
> > continue;
> > }
>
> Why are we handling this inside the loop?
Removed in v3.
> > + if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
> > + {
> > + /*
> > + * XXX What if we just came from root item, which is a plain heap
> > + * tuple? Do we need assert that notices when we reach an LP_DEAD
> > + * or LP_UNUSED item having started from such a root item?
> > + */
> > + Assert(!rootisredirect || nchain > 1);
>
> I don't think we can assert that.
Agreed, it has been removed in v3.
> I don't understand these new assert? We just set tupdead to false above?
I don't either. Removed in v3.
> I now actually wonder why this is correct. There's another comment about it:
>
> > We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
> > * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
>
> But that doesn't justify very much.
>
> What prevents the scenario that some other backend e.g. has a snapshot with
> xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
> than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
> row, but *not* the RECENTLY_DEAD one.
I'll need to think about this very carefully. I didn't think it was
worth blocking v3 on, though naturally it's a big concern.
> This comment can't really be understood without the historical behaviour of
> the function.
Agreed. Removed in v3.
> > + if (res == HEAPTUPLE_DEAD || res == HEAPTUPLE_RECENTLY_DEAD)
> > + {
> > + heap_prune_record_unused(prstate, offnum);
> > + HeapTupleHeaderAdvanceLatestRemovedXid(htup,
> > + &prstate->latestRemovedXid);
> > + }
> > + else
> > + Assert(false);
> It seems weird to have the if (res == HEAPTUPLE_DEAD ..) branch, but then to
> unconditionally return 1.
I changed that in v3.
> I'm not actually sure the Assert is unreachable. I can imagine cases where
> we'd see e.g. DELETE/INSERT_IN_PROGRESS due to a concurrent subtransaction
> abort or such.
I don't know either. I am leaving this question unresolved for now.
--
Peter Geoghegan
Attachments:
v3-0001-Fix-aborted-HOT-update-bug-in-heap-pruning.patch (application/octet-stream, 20.0K) download
From fd6025fd02f5df086ac4e4d29f17526e56dddf1f Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 8 Nov 2021 22:58:14 -0800
Subject: [PATCH v3] Fix aborted HOT update bug in heap pruning.
Author: Peter Geoghegan <pg@bowt.ie>
Reported-By: Alexander Lakhin <exclusion@gmail.com>
Diagnosed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Andres Freund <andres@anarazel.de>
Bug: #17255
Discussion: https://postgr.es/m/17255-14c0ac58d0f9b583@postgresql.org
Backpatch: 14, where snapshot scalability improvements first appear
---
src/backend/access/heap/pruneheap.c | 326 +++++++++++++++++-----------
1 file changed, 202 insertions(+), 124 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 50ed76198..79fe2bbf0 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -31,6 +31,7 @@
typedef struct
{
Relation rel;
+ BlockNumber targetblkno;
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
@@ -56,14 +57,16 @@ typedef struct
OffsetNumber redirected[MaxHeapTuplesPerPage * 2];
OffsetNumber nowdead[MaxHeapTuplesPerPage];
OffsetNumber nowunused[MaxHeapTuplesPerPage];
- /* marked[i] is true if item i is entered in one of the above arrays */
- bool marked[MaxHeapTuplesPerPage + 1];
+ /* visited[i] is true if item i is from a known valid HOT chain */
+ bool visited[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate);
+static int heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -249,24 +252,29 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.targetblkno = BufferGetBlockNumber(buffer);
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
prstate.latestRemovedXid = InvalidTransactionId;
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
- memset(prstate.marked, 0, sizeof(prstate.marked));
+ memset(prstate.visited, 0, sizeof(prstate.visited));
- /* Scan the page */
+ /*
+ * Scan the page to try to visit all items.
+ *
+ * It's possible that a few heap-only tuples will not get visited during
+ * our initial scan over the page. This happens when the tuples cannot be
+ * located by following a valid HOT chain.
+ */
maxoff = PageGetMaxOffsetNumber(page);
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- ItemId itemid;
-
- /* Ignore items already processed as part of an earlier chain */
- if (prstate.marked[offnum])
+ /* Ignore items already visited as part of an earlier chain */
+ if (prstate.visited[offnum])
continue;
/*
@@ -276,15 +284,33 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- /* Nothing to do if slot is empty or already dead */
- itemid = PageGetItemId(page, offnum);
- if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
- continue;
-
/* Process this item or chain of items */
ndeleted += heap_prune_chain(buffer, offnum, &prstate);
}
+ /*
+ * Scan the page again, processing heap-only tuples missed by first scan.
+ *
+ * These disconnected heap-only tuples (which always originate in aborted
+ * transactions) are always considered DEAD (and always become LP_UNUSED).
+ * Delaying processing of these tuples until our second pass allows our
+ * first pass to cleanly process entire HOT chains as whole units.
+ */
+ for (offnum = FirstOffsetNumber;
+ offnum <= maxoff;
+ offnum = OffsetNumberNext(offnum))
+ {
+ /* Ignore items already visited (only want disconnected tuples) */
+ if (prstate.visited[offnum])
+ continue;
+
+ if (off_loc)
+ *off_loc = offnum;
+
+ /* Process this disconnected heap-only tuple */
+ ndeleted += heap_prune_disconnected(buffer, offnum, &prstate);
+ }
+
/* Clear the offset information once we have processed the given page. */
if (off_loc)
*off_loc = InvalidOffsetNumber;
@@ -473,20 +499,21 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
- * Prune specified line pointer or a HOT chain originating at line pointer.
+ * Prune specified HOT chain originating at line pointer, or simple tuple.
*
- * If the item is an index-referenced tuple (i.e. not a heap-only tuple),
- * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
- * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
- * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
- * DEAD, the heap_prune_satisfies_vacuum test is just too coarse to detect it.
+ * Used during initial pass over the heap page. Won't process any heap-only
+ * tuples that cannot be found by traversing a HOT chain whose root item is at
+ * offset rootoffnum. Remaining "disconnected" heap-only tuples are dealt
+ * with in caller's second pass over the page instead. This is how we make
+ * sure that no DEAD tuples (or whole HOT chains) are missed during pruning.
*
* In general, pruning must never leave behind a DEAD tuple that still has
* tuple storage. VACUUM isn't prepared to deal with that case. That's why
* VACUUM prunes the same heap page a second time (without dropping its lock
* in the interim) when it sees a newly DEAD tuple that we initially saw as
- * in-progress. Retrying pruning like this can only happen when an inserting
- * transaction concurrently aborts.
+ * in-progress. Retrying pruning like this can only happen due to certain
+ * edge-cases, like the case where an inserting transaction concurrently
+ * aborts.
*
* The root line pointer is redirected to the tuple immediately after the
* latest DEAD tuple. If all tuples in the chain are DEAD, the root line
@@ -508,77 +535,59 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
{
- int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
TransactionId priorXmax = InvalidTransactionId;
ItemId rootlp;
- HeapTupleHeader htup;
OffsetNumber latestdead = InvalidOffsetNumber,
maxoff = PageGetMaxOffsetNumber(dp),
offnum;
OffsetNumber chainitems[MaxHeapTuplesPerPage];
- int nchain = 0,
- i;
- HeapTupleData tup;
+ int nchain;
- tup.t_tableOid = RelationGetRelid(prstate->rel);
+ Assert(!prstate->visited[rootoffnum]);
rootlp = PageGetItemId(dp, rootoffnum);
- /*
- * If it's a heap-only tuple, then it is not the start of a HOT chain.
- */
if (ItemIdIsNormal(rootlp))
{
- htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
-
- tup.t_data = htup;
- tup.t_len = ItemIdGetLength(rootlp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), rootoffnum);
+ HeapTupleHeader htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
+ /*
+ * If it's a heap-only tuple, then it is not the start of a HOT chain.
+ * We'll process it later, either by traversing its HOT chain
+ * (starting from the root item), or in heap_prune_disconnected() call
+ * made during our second pass over page.
+ */
if (HeapTupleHeaderIsHeapOnly(htup))
- {
- /*
- * If the tuple is DEAD and doesn't chain to anything else, mark
- * it unused immediately. (If it does chain, we can only remove
- * it as part of pruning its chain.)
- *
- * We need this primarily to handle aborted HOT updates, that is,
- * XMIN_INVALID heap-only tuples. Those might not be linked to by
- * any chain, since the parent tuple might be re-updated before
- * any pruning occurs. So we have to be able to reap them
- * separately from chain-pruning. (Note that
- * HeapTupleHeaderIsHotUpdated will never return true for an
- * XMIN_INVALID tuple, so this code will work even when there were
- * sequential updates within the aborted transaction.)
- *
- * Note that we might first arrive at a dead heap-only tuple
- * either here or while following a chain below. Whichever path
- * gets there first will mark the tuple unused.
- */
- if (heap_prune_satisfies_vacuum(prstate, &tup, buffer)
- == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
- {
- heap_prune_record_unused(prstate, rootoffnum);
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- ndeleted++;
- }
+ return 0;
+ }
+ else if (!ItemIdIsRedirected(rootlp))
+ {
+ /*
+ * Nothing to do if slot cannot possibly be valid root item of HOT
+ * chain or a simple heap tuple
+ */
+ Assert(ItemIdIsDead(rootlp) || !ItemIdIsUsed(rootlp));
+ prstate->visited[rootoffnum] = true;
- /* Nothing more to do */
- return ndeleted;
- }
+ return 0;
}
- /* Start from the root tuple */
+ /*
+ * Start from the root item. Mark it as valid up front, since root items
+ * are always processed up front, in first pass over page.
+ */
+ prstate->visited[rootoffnum] = true;
offnum = rootoffnum;
+ nchain = 0;
/* while not end of the chain */
for (;;)
{
ItemId lp;
- bool tupdead,
- recent_dead;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ bool tupdead;
/* Sanity check (pure paranoia) */
if (offnum < FirstOffsetNumber)
@@ -592,15 +601,11 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/* If item is already processed, stop --- it must not be same chain */
- if (prstate->marked[offnum])
+ if (nchain != 0 && prstate->visited[offnum])
break;
lp = PageGetItemId(dp, offnum);
- /* Unused item obviously isn't part of the chain */
- if (!ItemIdIsUsed(lp))
- break;
-
/*
* If we are looking at the redirected root line pointer, jump to the
* first normal tuple in the chain. If we find a redirect somewhere
@@ -615,20 +620,24 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
continue;
}
- /*
- * Likewise, a dead line pointer can't be part of the chain. (We
- * already eliminated the case of dead root tuple outside this
- * function.)
- */
- if (ItemIdIsDead(lp))
+ /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
+ if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
+ {
+ /*
+ * Can consider LP_UNUSED/LP_DEAD items visited when we arrive
+ * here by following a heap-only tuple's t_ctid link
+ */
+ prstate->visited[offnum] = true;
break;
+ }
Assert(ItemIdIsNormal(lp));
htup = (HeapTupleHeader) PageGetItem(dp, lp);
- tup.t_data = htup;
tup.t_len = ItemIdGetLength(lp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), offnum);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
/*
* Check the tuple XMIN against prior XMAX, if any
@@ -638,14 +647,18 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/*
- * OK, this tuple is indeed a member of the chain.
+ * OK, this tuple is indeed a member of the chain. Commit to this
+ * interpretation by marking the item as visited now.
*/
+ Assert((nchain == 0 && ItemIdIsNormal(rootlp)) ||
+ HeapTupleHeaderIsHeapOnly(htup));
chainitems[nchain++] = offnum;
+ prstate->visited[offnum] = true;
/*
* Check tuple's visibility status.
*/
- tupdead = recent_dead = false;
+ tupdead = false;
switch (heap_prune_satisfies_vacuum(prstate, &tup, buffer))
{
@@ -654,7 +667,6 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
case HEAPTUPLE_RECENTLY_DEAD:
- recent_dead = true;
/*
* This tuple may soon become DEAD. Update the hint field so
@@ -691,11 +703,9 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * Remember the last DEAD tuple seen. We will advance past
- * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
- * but we can't advance past anything else. We have to make sure that
- * we don't miss any DEAD tuples, since DEAD tuples that still have
- * tuple storage after pruning will confuse VACUUM.
+ * Remember the offnum of the last DEAD tuple in this HOT chain. We
+ * consider items up to and including the last such item we reach to
+ * be DEAD during final processing for HOT chain.
*/
if (tupdead)
{
@@ -703,12 +713,13 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
HeapTupleHeaderAdvanceLatestRemovedXid(htup,
&prstate->latestRemovedXid);
}
- else if (!recent_dead)
- break;
/*
* If the tuple is not HOT-updated, then we are at the end of this
- * HOT-update chain.
+ * HOT-update chain. There might actually be more tuples that were
+ * considered part of the same HOT chain in the past, before the
+ * updater's xact aborted. We'll process any such tuples later on
+ * instead, inside heap_prune_disconnected().
*/
if (!HeapTupleHeaderIsHotUpdated(htup))
break;
@@ -720,23 +731,34 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* Advance to next chain member.
*/
Assert(ItemPointerGetBlockNumber(&htup->t_ctid) ==
- BufferGetBlockNumber(buffer));
+ prstate->targetblkno);
offnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
}
/*
- * If we found a DEAD tuple in the chain, adjust the HOT chain so that all
- * the DEAD tuples at the start of the chain are removed and the root line
- * pointer is appropriately redirected.
+ * Should never find an LP_REDIRECT root item that didn't already point to
+ * a valid item. While a heap-only tuple's t_ctid link can legitimately
+ * point to almost anything, the rules for LP_REDIRECT links are far
+ * stricter: LP_REDIRECTs must _always_ point to a valid heap-only tuple.
*/
+ Assert(ItemIdIsNormal(rootlp) || nchain >= 2);
+
if (OffsetNumberIsValid(latestdead))
{
+ int i,
+ ndeleted = 0;
+
/*
- * Mark as unused each intermediate item that we are able to remove
- * from the chain.
+ * Okay, at least one tuple from the chain (or the single plain heap
+ * tuple) is considered DEAD. Record what to do with items in the
+ * chain now.
*
- * When the previous item is the last dead tuple seen, we are at the
+ * First deal with the non-root items from HOT chain. Mark earlier
+ * items we consider DEAD as LP_UNUSED (since they're heap-only
+ * tuples).
+ *
+ * When the previous item is the last DEAD tuple seen, we are at the
* right candidate for redirection.
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
@@ -746,36 +768,100 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * If the root entry had been a normal tuple, we are deleting it, so
- * count it in the result. But changing a redirect (even to DEAD
- * state) doesn't count.
+ * If the root item is a normal tuple, we are logically deleting it,
+ * so count it in the result. But changing an LP_REDIRECT (even to
+ * make it LP_DEAD) doesn't get counted in ndeleted -- that would
+ * amount to double-counting DEAD tuples (with tuple storage) in
+ * ndeleted.
*/
if (ItemIdIsNormal(rootlp))
ndeleted++;
/*
+ * Finally, consider what to do with the root item itself.
+ *
* If the DEAD tuple is at the end of the chain, the entire chain is
- * dead and the root line pointer can be marked dead. Otherwise just
- * redirect the root to the correct chain member.
+ * considered DEAD. The root line pointer must therefore become
+ * LP_DEAD. Otherwise just redirect the root to the correct chain
+ * member.
*/
if (i >= nchain)
heap_prune_record_dead(prstate, rootoffnum);
else
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
- }
- else if (nchain < 2 && ItemIdIsRedirected(rootlp))
- {
- /*
- * We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state.
- */
- heap_prune_record_dead(prstate, rootoffnum);
+
+ return ndeleted;
}
- return ndeleted;
+ return 0;
+}
+
+/*
+ * Handle disconnected heap-only tuples during second pass over page. We
+ * always expect to process these tuples as DEAD tuples here. Since they're
+ * heap-only tuples it follows that they'll always be set LP_UNUSED.
+ *
+ * This is how we handle aborted heap-only tuples that were not visited in our
+ * first pass (via HOT chain traversal with the usual cross-checks). These
+ * tuples occur when a parent tuple is updated, the updater aborts, and some
+ * unrelated updater re-updates the original parent tuple a second time. The
+ * parent's t_ctid link won't continue to point to the aborted tuple. (Even
+ * when it does, we won't consider the parent to have been HOT updated, just
+ * because its XMAX aborted -- so we still end up here for the aborted tuple).
+ *
+ * Like heap_prune_chain, we don't actually change the page here.
+ *
+ * Returns the number of tuples (to be) deleted from the page, though this
+ * should always be 1 in practice.
+*/
+static int
+heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate)
+{
+ Page dp = (Page) BufferGetPage(buffer);
+ ItemId lp;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ HTSV_Result res;
+
+ lp = PageGetItemId(dp, offnum);
+ Assert(ItemIdIsNormal(lp));
+ htup = (HeapTupleHeader) PageGetItem(dp, lp);
+
+ /*
+ * Caller must make sure that the tuple at 'offnum' is in fact a heap-only
+ * tuple that is disconnected from its HOT chain
+ */
+ Assert(!prstate->visited[offnum]);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+
+ /*
+ * We expect that disconnected heap-only tuples must be from aborted
+ * transactions. Any RECENTLY_DEAD tuples we see here are really DEAD,
+ * but the heap_prune_satisfies_vacuum test is too coarse to detect it.
+ */
+ tup.t_len = ItemIdGetLength(lp);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
+ res = heap_prune_satisfies_vacuum(prstate, &tup, buffer);
+ /* FIXME what do we actually not expect here? */
+ if (res != HEAPTUPLE_LIVE)
+ {
+ heap_prune_record_unused(prstate, offnum);
+
+ /* Unnecessary, but be conservative here */
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ return 1;
+ }
+
+ /*
+ * Should always be DEAD. A DEAD heap-only tuple is always counted in
+ * top-level ndeleted counter for pruning operation.
+ */
+ Assert(false);
+ return 0;
}
/* Record lowest soon-prunable XID */
@@ -801,10 +887,6 @@ heap_prune_record_redirect(PruneState *prstate,
prstate->redirected[prstate->nredirected * 2] = offnum;
prstate->redirected[prstate->nredirected * 2 + 1] = rdoffnum;
prstate->nredirected++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
- Assert(!prstate->marked[rdoffnum]);
- prstate->marked[rdoffnum] = true;
}
/* Record line pointer to be marked dead */
@@ -814,8 +896,6 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
/* Record line pointer to be marked unused */
@@ -825,8 +905,6 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->nunused < MaxHeapTuplesPerPage);
prstate->nowunused[prstate->nunused] = offnum;
prstate->nunused++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 04:22 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 04:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Nov 11, 2021 at 4:58 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > What prevents the scenario that some other backend e.g. has a snapshot with
> > xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
> > than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
> > row, but *not* the RECENTLY_DEAD one.
>
> I'll need to think about this very carefully. I didn't think it was
> worth blocking v3 on, though naturally it's a big concern.
If we're to traverse HOT chains right to the end in
heap_prune_chain(), reading even LIVE tuples (per the approach
proposed in my bugfix patch), we probably need to be more careful
about concurrently aborted xacts -- relying on the usual
!HeapTupleHeaderIsHotUpdated(htup) test doesn't seem safe.
Imagine if we land on a concurrently-aborted DEAD tuple at the end of
a physical HOT chain -- this might not be caught before we test the
previous tuple in the chain using HeapTupleHeaderIsHotUpdated(htup) --
the abort might happen just as we land on the final/aborted tuple. We
certainly shouldn't conclude that the whole HOT chain is now DEAD,
just because that one tuple is dead.
That definitely cannot happen on HEAD, I think, because we just give
up as soon as we see anything that isn't either DEAD or RECENTLY_DEAD.
But maybe it's possible with the patch.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 05:46 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 05:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Nov 11, 2021 at 8:22 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, Nov 11, 2021 at 4:58 PM Peter Geoghegan <pg@bowt.ie> wrote:
> > > What prevents the scenario that some other backend e.g. has a snapshot with
> > > xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
> > > than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
> > > row, but *not* the RECENTLY_DEAD one.
> >
> > I'll need to think about this very carefully. I didn't think it was
> > worth blocking v3 on, though naturally it's a big concern.
>
> If we're to traverse HOT chains right to the end in
> heap_prune_chain(), reading even LIVE tuples (per the approach
> proposed in my bugfix patch), we probably need to be more careful
> about concurrently aborted xacts -- relying on the usual
> !HeapTupleHeaderIsHotUpdated(htup) test doesn't seem safe.
I wonder if we're approaching this business with "RECENTLY_DEAD can be
upgraded to DEAD" in entirely the wrong way. Why not just not do that
at all anymore, on the off chance that it's unsafe? Why even take a
small chance? Our decision has to work at the level of the whole
entire HOT chain, and it seems to me that we should make that as
simple as possible.
I'm pretty sure that we're giving up nothing this way. We can just
take the conservative position that it's never okay for
heap_prune_chain() to consider a heap-only tuple from a validated HOT
chain DEAD, unless A.) HTSV says it is DEAD when asked, and B.) HTSV
said the same thing (tuple DEAD) about any and all earlier heap-only
tuples in the chain. Moreover, if HTSV says
LIVE/RECENTLY_DEAD/whatever about one tuple in the chain, then we
refuse to treat *ALL* successor tuples in the same chain as DEAD --
even when HTSV says that they're DEAD directly. As long as these
DEAD-to-HTSV heap-only tuples appear to be from the same original HOT
chain, we don't need to change our mind about the HOT chain.
The original structure of heap_prune_chain() from the HOT commit in
2007 had more or less the same code and structure as it has now, but
almost everything was in a critical structure -- the state arrays and
so on only came about a year later, in commit 6f10eb2111. The proposed
bug fix more or less finishes the work of the second commit, which
didn't go far enough. As long as we are starting by building a
consistent picture of valid HOT chains on the page, and only later
handle disconnected heap-only tuples, everything works out.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 21:11 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 21:11 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Nov 11, 2021 at 9:46 PM Peter Geoghegan <pg@bowt.ie> wrote:
> I wonder if we're approaching this business with "RECENTLY_DEAD can be
> upgraded to DEAD" in entirely the wrong way. Why not just not do that
> at all anymore, on the off chance that it's unsafe? Why even take a
> small chance? Our decision has to work at the level of the whole
> entire HOT chain, and it seems to me that we should make that as
> simple as possible.
Attached revision does it that way.
It also addresses the separate issue of DEAD vs RECENTLY_DEAD
disconnected tuples -- that was the other unresolved question. This
revision takes a harder line on the state of disconnected heap-only
tuples. Andres said that he doesn't know for sure that disconnected
heap-only tuples cannot be DELETE/INSERT_IN_PROGRESS -- "I'm not
actually sure the Assert is unreachable. I can imagine cases where
we'd see e.g. DELETE/INSERT_IN_PROGRESS due to a concurrent
subtransaction abort or such". But I don't see how that's possible. In
fact, I don't even see how it's possible for these items to be
RECENTLY_DEAD -- I think that they must always be DEAD (or we're in
big trouble anyway).
These are not just any heap-only tuples. They're heap-only tuples that
cannot possibly be accessed from a HOT chain. And so it's just
physically impossible for them to be returned by index scans -- this
is a certainty. How could they not be DEAD, in every possible sense?
How could they not come from an aborted transaction, specifically?
Naturally, I also went through the exercise of trying to find a
counterexample, where pruning doesn't see a disconnected tuple as DEAD
in its HTSV. I could not get the assertion to fail with Alexander's
test case, nor with make check-world. If the assertion did fail, then
I imagine that that would have to be due to a problem elsewhere -- it
couldn't be a problem with the "disconnected heap-only tuples must
already be DEAD to HTSV" assumption itself. That is one of the few
things about this patch that *isn't* complicated.
--
Peter Geoghegan
Attachments:
v4-0001-Fix-aborted-HOT-update-bug-in-heap-pruning.patch (application/octet-stream, 21.0K) download
From 3f7fcf89c81eb38982dcac3ada038e6f3a539090 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 8 Nov 2021 22:58:14 -0800
Subject: [PATCH v4] Fix aborted HOT update bug in heap pruning.
Author: Peter Geoghegan <pg@bowt.ie>
Reported-By: Alexander Lakhin <exclusion@gmail.com>
Diagnosed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Andres Freund <andres@anarazel.de>
Bug: #17255
Discussion: https://postgr.es/m/17255-14c0ac58d0f9b583@postgresql.org
Backpatch: 14, where snapshot scalability improvements first appear
---
src/backend/access/heap/pruneheap.c | 361 +++++++++++++++++-----------
1 file changed, 218 insertions(+), 143 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 50ed76198..0bbef4647 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -31,6 +31,7 @@
typedef struct
{
Relation rel;
+ BlockNumber targetblkno;
/* tuple visibility test, initialized for the relation */
GlobalVisState *vistest;
@@ -56,14 +57,16 @@ typedef struct
OffsetNumber redirected[MaxHeapTuplesPerPage * 2];
OffsetNumber nowdead[MaxHeapTuplesPerPage];
OffsetNumber nowunused[MaxHeapTuplesPerPage];
- /* marked[i] is true if item i is entered in one of the above arrays */
- bool marked[MaxHeapTuplesPerPage + 1];
+ /* visited[i] is true if item i is from a known valid HOT chain */
+ bool visited[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate);
+static int heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -249,24 +252,29 @@ heap_page_prune(Relation relation, Buffer buffer,
*/
prstate.new_prune_xid = InvalidTransactionId;
prstate.rel = relation;
+ prstate.targetblkno = BufferGetBlockNumber(buffer);
prstate.vistest = vistest;
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
prstate.latestRemovedXid = InvalidTransactionId;
prstate.nredirected = prstate.ndead = prstate.nunused = 0;
- memset(prstate.marked, 0, sizeof(prstate.marked));
+ memset(prstate.visited, 0, sizeof(prstate.visited));
- /* Scan the page */
+ /*
+ * Scan the page to try to visit all items.
+ *
+ * It's possible that a few heap-only tuples will not get visited during
+ * our initial scan over the page. This happens when the tuples cannot be
+ * located by following a valid HOT chain.
+ */
maxoff = PageGetMaxOffsetNumber(page);
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- ItemId itemid;
-
- /* Ignore items already processed as part of an earlier chain */
- if (prstate.marked[offnum])
+ /* Ignore items already visited as part of an earlier chain */
+ if (prstate.visited[offnum])
continue;
/*
@@ -276,15 +284,33 @@ heap_page_prune(Relation relation, Buffer buffer,
if (off_loc)
*off_loc = offnum;
- /* Nothing to do if slot is empty or already dead */
- itemid = PageGetItemId(page, offnum);
- if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
- continue;
-
/* Process this item or chain of items */
ndeleted += heap_prune_chain(buffer, offnum, &prstate);
}
+ /*
+ * Scan the page again, processing heap-only tuples missed by first scan.
+ *
+ * These disconnected heap-only tuples (which always originate in aborted
+ * transactions) are always considered DEAD (and always become LP_UNUSED).
+ * Delaying processing of these tuples until our second pass allows our
+ * first pass to cleanly process entire HOT chains as whole units.
+ */
+ for (offnum = FirstOffsetNumber;
+ offnum <= maxoff;
+ offnum = OffsetNumberNext(offnum))
+ {
+ /* Ignore items already visited (only want disconnected tuples) */
+ if (prstate.visited[offnum])
+ continue;
+
+ if (off_loc)
+ *off_loc = offnum;
+
+ /* Process this disconnected heap-only tuple */
+ ndeleted += heap_prune_disconnected(buffer, offnum, &prstate);
+ }
+
/* Clear the offset information once we have processed the given page. */
if (off_loc)
*off_loc = InvalidOffsetNumber;
@@ -473,20 +499,21 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
- * Prune specified line pointer or a HOT chain originating at line pointer.
+ * Prune specified HOT chain originating at line pointer, or simple tuple.
*
- * If the item is an index-referenced tuple (i.e. not a heap-only tuple),
- * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
- * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
- * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
- * DEAD, the heap_prune_satisfies_vacuum test is just too coarse to detect it.
+ * Used during initial pass over the heap page. Won't process any heap-only
+ * tuples that cannot be found by traversing a HOT chain whose root item is at
+ * offset rootoffnum. Remaining "disconnected" heap-only tuples are dealt
+ * with in caller's second pass over the page instead. This is how we make
+ * sure that no DEAD tuples (or whole HOT chains) are missed during pruning.
*
* In general, pruning must never leave behind a DEAD tuple that still has
* tuple storage. VACUUM isn't prepared to deal with that case. That's why
* VACUUM prunes the same heap page a second time (without dropping its lock
* in the interim) when it sees a newly DEAD tuple that we initially saw as
- * in-progress. Retrying pruning like this can only happen when an inserting
- * transaction concurrently aborts.
+ * in-progress. Retrying pruning like this can only happen due to certain
+ * edge-cases, like the case where an inserting transaction concurrently
+ * aborts.
*
* The root line pointer is redirected to the tuple immediately after the
* latest DEAD tuple. If all tuples in the chain are DEAD, the root line
@@ -508,77 +535,59 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
{
- int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
TransactionId priorXmax = InvalidTransactionId;
ItemId rootlp;
- HeapTupleHeader htup;
OffsetNumber latestdead = InvalidOffsetNumber,
maxoff = PageGetMaxOffsetNumber(dp),
offnum;
+ bool pastlatestdead = false;
OffsetNumber chainitems[MaxHeapTuplesPerPage];
- int nchain = 0,
- i;
- HeapTupleData tup;
+ int nchain;
- tup.t_tableOid = RelationGetRelid(prstate->rel);
+ Assert(!prstate->visited[rootoffnum]);
rootlp = PageGetItemId(dp, rootoffnum);
- /*
- * If it's a heap-only tuple, then it is not the start of a HOT chain.
- */
if (ItemIdIsNormal(rootlp))
{
- htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
-
- tup.t_data = htup;
- tup.t_len = ItemIdGetLength(rootlp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), rootoffnum);
+ HeapTupleHeader htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
+ /*
+ * If it's a heap-only tuple, then it is not the start of a HOT chain.
+ * We'll process it later, either by traversing its HOT chain
+ * (starting from the root item), or in heap_prune_disconnected() call
+ * made during our second pass over page.
+ */
if (HeapTupleHeaderIsHeapOnly(htup))
- {
- /*
- * If the tuple is DEAD and doesn't chain to anything else, mark
- * it unused immediately. (If it does chain, we can only remove
- * it as part of pruning its chain.)
- *
- * We need this primarily to handle aborted HOT updates, that is,
- * XMIN_INVALID heap-only tuples. Those might not be linked to by
- * any chain, since the parent tuple might be re-updated before
- * any pruning occurs. So we have to be able to reap them
- * separately from chain-pruning. (Note that
- * HeapTupleHeaderIsHotUpdated will never return true for an
- * XMIN_INVALID tuple, so this code will work even when there were
- * sequential updates within the aborted transaction.)
- *
- * Note that we might first arrive at a dead heap-only tuple
- * either here or while following a chain below. Whichever path
- * gets there first will mark the tuple unused.
- */
- if (heap_prune_satisfies_vacuum(prstate, &tup, buffer)
- == HEAPTUPLE_DEAD && !HeapTupleHeaderIsHotUpdated(htup))
- {
- heap_prune_record_unused(prstate, rootoffnum);
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- ndeleted++;
- }
+ return 0;
+ }
+ else if (!ItemIdIsRedirected(rootlp))
+ {
+ /*
+ * Nothing to do if slot cannot possibly be valid root item of HOT
+ * chain or a simple heap tuple
+ */
+ Assert(ItemIdIsDead(rootlp) || !ItemIdIsUsed(rootlp));
+ prstate->visited[rootoffnum] = true;
- /* Nothing more to do */
- return ndeleted;
- }
+ return 0;
}
- /* Start from the root tuple */
+ /*
+ * Start from the root item. Mark it as valid up front, since root items
+ * are always processed up front, in first pass over page.
+ */
+ prstate->visited[rootoffnum] = true;
offnum = rootoffnum;
+ nchain = 0;
/* while not end of the chain */
for (;;)
{
ItemId lp;
- bool tupdead,
- recent_dead;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
/* Sanity check (pure paranoia) */
if (offnum < FirstOffsetNumber)
@@ -592,15 +601,11 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/* If item is already processed, stop --- it must not be same chain */
- if (prstate->marked[offnum])
+ if (nchain != 0 && prstate->visited[offnum])
break;
lp = PageGetItemId(dp, offnum);
- /* Unused item obviously isn't part of the chain */
- if (!ItemIdIsUsed(lp))
- break;
-
/*
* If we are looking at the redirected root line pointer, jump to the
* first normal tuple in the chain. If we find a redirect somewhere
@@ -615,20 +620,24 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
continue;
}
- /*
- * Likewise, a dead line pointer can't be part of the chain. (We
- * already eliminated the case of dead root tuple outside this
- * function.)
- */
- if (ItemIdIsDead(lp))
+ /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
+ if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
+ {
+ /*
+ * Can consider LP_UNUSED/LP_DEAD items visited when we arrive
+ * here by following a heap-only tuple's t_ctid link
+ */
+ prstate->visited[offnum] = true;
break;
+ }
Assert(ItemIdIsNormal(lp));
htup = (HeapTupleHeader) PageGetItem(dp, lp);
- tup.t_data = htup;
tup.t_len = ItemIdGetLength(lp);
- ItemPointerSet(&(tup.t_self), BufferGetBlockNumber(buffer), offnum);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
/*
* Check the tuple XMIN against prior XMAX, if any
@@ -638,33 +647,46 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/*
- * OK, this tuple is indeed a member of the chain.
+ * OK, this tuple is indeed a member of the chain. Stick with this
+ * interpretation for the entire prune operation by marking the item
+ * as visited now.
*/
+ Assert((nchain == 0 && ItemIdIsNormal(rootlp)) ||
+ HeapTupleHeaderIsHeapOnly(htup));
chainitems[nchain++] = offnum;
+ prstate->visited[offnum] = true;
/*
* Check tuple's visibility status.
*/
- tupdead = recent_dead = false;
-
switch (heap_prune_satisfies_vacuum(prstate, &tup, buffer))
{
case HEAPTUPLE_DEAD:
- tupdead = true;
- break;
-
- case HEAPTUPLE_RECENTLY_DEAD:
- recent_dead = true;
/*
- * This tuple may soon become DEAD. Update the hint field so
- * that the page is reconsidered for pruning in future.
+ * Remember the offnum of the last DEAD tuple in this HOT
+ * chain. To keep things simple, don't treat heap-only tuples
+ * from a HOT chain as DEAD unless they're only preceded by
+ * other DEAD tuples (in addition to actually being DEAD).
+ * Remaining tuples that appear DEAD (but don't get treated as
+ * such by us) are from concurrently aborting updaters.
+ *
+ * VACUUM will ask us to prune the heap page a second time
+ * when it sees that there is a DEAD tuple left behind, but
+ * that would be necessary regardless of our approach here.
*/
- heap_prune_record_prunable(prstate,
- HeapTupleHeaderGetUpdateXid(htup));
- break;
+ if (!pastlatestdead)
+ {
+ latestdead = offnum;
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ break;
+ }
+ /* FALL THRU */
+ case HEAPTUPLE_RECENTLY_DEAD:
case HEAPTUPLE_DELETE_IN_PROGRESS:
+ pastlatestdead = true; /* no further DEAD tuples in CHAIN */
/*
* This tuple may soon become DEAD. Update the hint field so
@@ -676,6 +698,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
case HEAPTUPLE_LIVE:
case HEAPTUPLE_INSERT_IN_PROGRESS:
+ pastlatestdead = true; /* no further DEAD tuples in CHAIN */
/*
* If we wanted to optimize for aborts, we might consider
@@ -690,25 +713,12 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
}
- /*
- * Remember the last DEAD tuple seen. We will advance past
- * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
- * but we can't advance past anything else. We have to make sure that
- * we don't miss any DEAD tuples, since DEAD tuples that still have
- * tuple storage after pruning will confuse VACUUM.
- */
- if (tupdead)
- {
- latestdead = offnum;
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- }
- else if (!recent_dead)
- break;
-
/*
* If the tuple is not HOT-updated, then we are at the end of this
- * HOT-update chain.
+ * HOT-update chain. There might actually be more tuples that were
+ * considered part of the same HOT chain in the past, before the
+ * updater's xact aborted. We'll process any such tuples later on
+ * instead, inside heap_prune_disconnected().
*/
if (!HeapTupleHeaderIsHotUpdated(htup))
break;
@@ -720,23 +730,35 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
* Advance to next chain member.
*/
Assert(ItemPointerGetBlockNumber(&htup->t_ctid) ==
- BufferGetBlockNumber(buffer));
+ prstate->targetblkno);
offnum = ItemPointerGetOffsetNumber(&htup->t_ctid);
priorXmax = HeapTupleHeaderGetUpdateXid(htup);
}
/*
- * If we found a DEAD tuple in the chain, adjust the HOT chain so that all
- * the DEAD tuples at the start of the chain are removed and the root line
- * pointer is appropriately redirected.
+ * Should never find an LP_REDIRECT root item that didn't already point to
+ * a valid item. While a heap-only tuple's t_ctid link can legitimately
+ * point to almost anything, the rules for LP_REDIRECT links are far
+ * stricter: LP_REDIRECTs must _always_ point to a valid heap-only tuple.
*/
+ Assert(ItemIdIsNormal(rootlp) ||
+ (ItemIdIsRedirected(rootlp) && nchain >= 2));
+
if (OffsetNumberIsValid(latestdead))
{
+ int i,
+ ndeleted = 0;
+
/*
- * Mark as unused each intermediate item that we are able to remove
- * from the chain.
+ * Okay, at least one tuple from the chain (or the single plain heap
+ * tuple) is considered DEAD. Record what to do with items in the
+ * chain now.
*
- * When the previous item is the last dead tuple seen, we are at the
+ * First deal with the non-root items from HOT chain. Mark earlier
+ * items we consider DEAD as LP_UNUSED (since they're heap-only
+ * tuples).
+ *
+ * When the previous item is the last DEAD tuple seen, we are at the
* right candidate for redirection.
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
@@ -746,36 +768,97 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * If the root entry had been a normal tuple, we are deleting it, so
- * count it in the result. But changing a redirect (even to DEAD
- * state) doesn't count.
+ * If the root item is a normal tuple, we are logically deleting it,
+ * so count it in the result. But changing an LP_REDIRECT (even to
+ * make it LP_DEAD) doesn't get counted in ndeleted -- that would
+ * amount to double-counting DEAD tuples (with tuple storage) in
+ * ndeleted.
*/
if (ItemIdIsNormal(rootlp))
ndeleted++;
/*
+ * Finally, consider what to do with the root item itself.
+ *
* If the DEAD tuple is at the end of the chain, the entire chain is
- * dead and the root line pointer can be marked dead. Otherwise just
- * redirect the root to the correct chain member.
+ * considered DEAD. The root item must therefore become LP_DEAD.
+ * Otherwise just redirect the root to the correct chain member.
*/
if (i >= nchain)
heap_prune_record_dead(prstate, rootoffnum);
else
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
- }
- else if (nchain < 2 && ItemIdIsRedirected(rootlp))
- {
- /*
- * We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state.
- */
- heap_prune_record_dead(prstate, rootoffnum);
+
+ return ndeleted;
}
- return ndeleted;
+ return 0;
+}
+
+/*
+ * Handle disconnected heap-only tuples during second pass over page. We
+ * always expect to process these tuples as DEAD tuples here. Since they're
+ * heap-only tuples it follows that they'll always be set LP_UNUSED.
+ *
+ * This is how we handle aborted heap-only tuples that were not visited in our
+ * first pass (via HOT chain traversal with the usual cross-checks). These
+ * tuples occur when a parent tuple is updated, the updater aborts, and some
+ * unrelated updater re-updates the original parent tuple a second time. The
+ * parent's t_ctid link won't continue to point to the aborted tuple. (Even
+ * when it does, we won't consider the parent to have been HOT updated, just
+ * because its XMAX aborted -- so we still end up here for the aborted tuple).
+ *
+ * Like heap_prune_chain, we don't actually change the page here.
+ *
+ * Returns the number of tuples (to be) deleted from the page, though this
+ * should always be 1 in practice.
+*/
+static int
+heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate)
+{
+ Page dp = (Page) BufferGetPage(buffer);
+ ItemId lp;
+ HeapTupleHeader htup;
+ HeapTupleData tup;
+ HTSV_Result res;
+
+ lp = PageGetItemId(dp, offnum);
+ Assert(ItemIdIsNormal(lp));
+ htup = (HeapTupleHeader) PageGetItem(dp, lp);
+
+ /*
+ * Caller must make sure that the tuple at 'offnum' is in fact a heap-only
+ * tuple that is disconnected from its HOT chain
+ */
+ Assert(!prstate->visited[offnum]);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+
+ /*
+ * We expect that disconnected heap-only tuples must be from aborted
+ * transactions. They must already be DEAD, or something is amiss.
+ */
+ tup.t_len = ItemIdGetLength(lp);
+ tup.t_tableOid = RelationGetRelid(prstate->rel);
+ tup.t_data = htup;
+ ItemPointerSet(&(tup.t_self), prstate->targetblkno, offnum);
+ res = heap_prune_satisfies_vacuum(prstate, &tup, buffer);
+ if (res == HEAPTUPLE_DEAD)
+ {
+ heap_prune_record_unused(prstate, offnum);
+
+ /* Unnecessary, but be conservative here */
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ return 1;
+ }
+
+ /*
+ * Should always be DEAD. A DEAD heap-only tuple is always counted in
+ * top-level ndeleted counter for pruning operation.
+ */
+ Assert(false);
+ return 0;
}
/* Record lowest soon-prunable XID */
@@ -801,10 +884,6 @@ heap_prune_record_redirect(PruneState *prstate,
prstate->redirected[prstate->nredirected * 2] = offnum;
prstate->redirected[prstate->nredirected * 2 + 1] = rdoffnum;
prstate->nredirected++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
- Assert(!prstate->marked[rdoffnum]);
- prstate->marked[rdoffnum] = true;
}
/* Record line pointer to be marked dead */
@@ -814,8 +893,6 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
/* Record line pointer to be marked unused */
@@ -825,8 +902,6 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->nunused < MaxHeapTuplesPerPage);
prstate->nowunused[prstate->nunused] = offnum;
prstate->nunused++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 22:29 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-12 22:29 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-12 13:11:54 -0800, Peter Geoghegan wrote:
> It also addresses the separate issue of DEAD vs RECENTLY_DEAD
> disconnected tuples -- that was the other unresolved question. This
> revision takes a harder line on the state of disconnected heap-only
> tuples. Andres said that he doesn't know for sure that disconnected
> heap-only tuples cannot be DELETE/INSERT_IN_PROGRESS -- "I'm not
> actually sure the Assert is unreachable. I can imagine cases where
> we'd see e.g. DELETE/INSERT_IN_PROGRESS due to a concurrent
> subtransaction abort or such". But I don't see how that's possible. In
> fact, I don't even see how it's possible for these items to be
> RECENTLY_DEAD -- I think that they must always be DEAD (or we're in
> big trouble anyway).
>
> These are not just any heap-only tuples. They're heap-only tuples that
> cannot possibly be accessed from a HOT chain. And so it's just
> physically impossible for them to be returned by index scans -- this
> is a certainty. How could they not be DEAD, in every possible sense?
> How could they not come from an aborted transaction, specifically?
With subtransactions abort is a bit more complicated than with plain
transactions. I'm not at all sure a problematic scenario exists, but I
wouldn't want to rely on it.
Especially if suboverflowed comes into play there can be scenarios where one
backend uses TransactionIdDidAbort() + SubTransGetTopmostTransaction() for
in-progress determination while another just relies on the procarray. Those
aren't updated atomically with respect to each other.
Also, heap_update()'s wait = true path uses a bit different logic again to
wait for other backends than what HeapTupleSatisfiesVacuum() ends up with.
> Naturally, I also went through the exercise of trying to find a
> counterexample, where pruning doesn't see a disconnected tuple as DEAD
> in its HTSV. I could not get the assertion to fail with Alexander's
> test case, nor with make check-world.
I don't think that provides a meaningful coverage. Alexander's test has a
quite limited set operations (which e.g. doesn't include an subxacts), and our
own tests around subtransactions, and particularly concurrent subtransaction
heavy work, is quite, uh, minimal.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 22:46 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 22:46 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 2:29 PM Andres Freund <andres@anarazel.de> wrote:
> With subtransactions abort is a bit more complicated than with plain
> transactions. I'm not at all sure a problematic scenario exists, but I
> wouldn't want to rely on it.
What would it actually mean to rely on it, or to not rely on it?
As I've pointed out many times already, a disconnected heap tuple
cannot be accessed from an index scan -- this is something that you
*can* rely on, because we've performed exactly the same steps as
heap_hot_search_buffer() would in making that determination. When you
talk about what HTSV thinks of the tuple, you're merely talking about
how to behave in the event of a specific form of HOT chain corruption
(a theoretical background risk for HOT chains that's nothing new).
This is a question of trade-offs around adding defensive checks and so
on. It is not a question of making the corruption itself any less
likely (unless early detection allows the user to prevent further
corruption). I'm a bit confused here, because it sounds like you might
not agree with that.
> > Naturally, I also went through the exercise of trying to find a
> > counterexample, where pruning doesn't see a disconnected tuple as DEAD
> > in its HTSV. I could not get the assertion to fail with Alexander's
> > test case, nor with make check-world.
>
> I don't think that provides a meaningful coverage. Alexander's test has a
> quite limited set operations (which e.g. doesn't include an subxacts), and our
> own tests around subtransactions, and particularly concurrent subtransaction
> heavy work, is quite, uh, minimal.
It's a start.
We need to be pragmatic here. There is some uncertainty about what
HTSV might say about a disconnected tuple in the absence of
corruption, or there is a risk of a new problem like that coming up in
the future -- let's work within those confines, then. What do you want
to do about that? There aren't that many choices, since, to repeat,
the tuple is "morally" DEAD no matter what. Even with corruption, even
without corruption in the presence of some unanticipated corner case
with HTSV -- this is fundamental.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 22:57 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-12 22:57 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-12 14:46:22 -0800, Peter Geoghegan wrote:
> On Fri, Nov 12, 2021 at 2:29 PM Andres Freund <andres@anarazel.de> wrote:
> > With subtransactions abort is a bit more complicated than with plain
> > transactions. I'm not at all sure a problematic scenario exists, but I
> > wouldn't want to rely on it.
>
> What would it actually mean to rely on it, or to not rely on it?
That we shouldn't throw an error / assert out if we find such a tuple.
> As I've pointed out many times already, a disconnected heap tuple
> cannot be accessed from an index scan -- this is something that you
> *can* rely on, because we've performed exactly the same steps as
> heap_hot_search_buffer() would in making that determination.
Yes, it'd also not be considered visible by SatisfiesMVCC().
> When you talk about what HTSV thinks of the tuple, you're merely talking
> about how to behave in the event of a specific form of HOT chain corruption
> (a theoretical background risk for HOT chains that's nothing new).
My point is that I don't think it necessarily signals corruption. But a very
short term transient state under heavy concurrency.
> We need to be pragmatic here. There is some uncertainty about what
> HTSV might say about a disconnected tuple in the absence of
> corruption, or there is a risk of a new problem like that coming up in
> the future -- let's work within those confines, then. What do you want
> to do about that? There aren't that many choices, since, to repeat,
> the tuple is "morally" DEAD no matter what. Even with corruption, even
> without corruption in the presence of some unanticipated corner case
> with HTSV -- this is fundamental.
I think we can assert/error out if it's visible, that's clearly
corruption. I'd personally not add assert/error checks for other states, given
that it could plausible happen without indicating a problem. Debugging
transient errors that happen rarely, under high load, with nontrivial
workloads isn't fun.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 23:12 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 23:12 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 2:57 PM Andres Freund <andres@anarazel.de> wrote:
> > What would it actually mean to rely on it, or to not rely on it?
>
> That we shouldn't throw an error / assert out if we find such a tuple.
I agree with you about making this an error -- let's not do that. But
I disagree about making this an assertion. I *strongly* disagree, in
fact.
I am quite willing to accept general uncertainty about what's really
possible when it comes to what HTSV might say in edge cases. But I
think we need to take a firm position on what we believe is possible,
by having an assertion that formally represents what we believe to be
true, including our reasoning. If that turns out to be wrong, great --
we then have the opportunity to be less wrong in the future. What has
it cost us? The minor annoyance of turning a buildfarm animal red for
a while?
> > When you talk about what HTSV thinks of the tuple, you're merely talking
> > about how to behave in the event of a specific form of HOT chain corruption
> > (a theoretical background risk for HOT chains that's nothing new).
>
> My point is that I don't think it necessarily signals corruption. But a very
> short term transient state under heavy concurrency.
I think that that's reasonable as a working assumption -- I really do.
I also think that you need pretty thorough assertions for this.
> > We need to be pragmatic here. There is some uncertainty about what
> > HTSV might say about a disconnected tuple in the absence of
> > corruption, or there is a risk of a new problem like that coming up in
> > the future -- let's work within those confines, then. What do you want
> > to do about that? There aren't that many choices, since, to repeat,
> > the tuple is "morally" DEAD no matter what. Even with corruption, even
> > without corruption in the presence of some unanticipated corner case
> > with HTSV -- this is fundamental.
>
> I think we can assert/error out if it's visible, that's clearly
> corruption. I'd personally not add assert/error checks for other states, given
> that it could plausible happen without indicating a problem. Debugging
> transient errors that happen rarely, under high load, with nontrivial
> workloads isn't fun.
What if it's just an assertion failure (just for non-LIVE HTSV return
codes)? An assertion is a very different thing to an defensive "can't
happen" ERROR.
There is a decent chance that there is a bug if the return code is,
say, HEAPTUPLE_INSERT_IN_PROGRESS. I just cannot think of a scenario
where this specific code path sees a tuple as INSERT_IN_PROGRESS that
isn't also very broken.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 23:31 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-12 23:31 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-12 13:11:54 -0800, Peter Geoghegan wrote:
> Attached revision does it that way.
I wonder if we should try to go for something considerably simpler for 14. How
about having a new array that just stores the HTSV state for every
ItemIdIsNormal(). For simplicity, we could populate that array eagerly in a
separate loop.
That'd fix the known bugs, and yield better efficiency (because we'd not
re-compute HTSV all the time). Then for HEAD go for something that fixes
pruning more fundamentally.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 23:47 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 23:47 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 3:31 PM Andres Freund <andres@anarazel.de> wrote:
> I wonder if we should try to go for something considerably simpler for 14. How
> about having a new array that just stores the HTSV state for every
> ItemIdIsNormal(). For simplicity, we could populate that array eagerly in a
> separate loop.
Why is that simpler than a boolean array, which represents whether or
not the item has had its heap_prune_record_unused() call yet (if it's
a tuple with storage)?
> That'd fix the known bugs, and yield better efficiency (because we'd not
> re-compute HTSV all the time). Then for HEAD go for something that fixes
> pruning more fundamentally.
I don't know what you mean about the patch recomputing HTSV all the
time. The patch doesn't do that.
It's true that we'll call HTSV (heap_prune_record_unused(), actually)
more often when following HOT chains, because now we follow them until
the end. However, the heap_prune_record_unused() calls only happen
after we've already validated that we found a heap-only tuple that's
part of the same HOT chain. That just leaves disconnected tuples. We
do call heap_prune_record_unused() there too (which is theoretically
unnecessary), but only once.
Overall, we are *guaranteed* to only call heap_prune_record_unused()
at most once per tuple with storage. I believe that this is a small
reduction, since HEAD will do the maybe-aborted precheck call to
heap_prune_record_unused() before anything else.
I guess you might have meant something about the more-conservative
behavior with DEAD vs RECENTLY_DEAD during HOT chain traversal. But
the cases where that makes any difference at all ought to be very rare
indeed.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 23:48 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-12 23:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 3:47 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Overall, we are *guaranteed* to only call heap_prune_record_unused()
> at most once per tuple with storage. I believe that this is a small
> reduction, since HEAD will do the maybe-aborted precheck call to
> heap_prune_record_unused() before anything else.
This is copy-pasta. I meant heap_prune_satisfies_vacuum(), not
heap_prune_record_unused().
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-12 23:56 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-12 23:56 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-12 15:47:45 -0800, Peter Geoghegan wrote:
> On Fri, Nov 12, 2021 at 3:31 PM Andres Freund <andres@anarazel.de> wrote:
> > I wonder if we should try to go for something considerably simpler for 14. How
> > about having a new array that just stores the HTSV state for every
> > ItemIdIsNormal(). For simplicity, we could populate that array eagerly in a
> > separate loop.
>
> Why is that simpler than a boolean array, which represents whether or
> not the item has had its heap_prune_record_unused() call yet (if it's
> a tuple with storage)?
Well, your change is basically a new approach of pruning - a much better
one. But it's a larger change than just eliminating the repeated HTSV calls so
that they cannot change over time. That'd be ~10-15 lines.
> > That'd fix the known bugs, and yield better efficiency (because we'd not
> > re-compute HTSV all the time). Then for HEAD go for something that fixes
> > pruning more fundamentally.
>
> I don't know what you mean about the patch recomputing HTSV all the
> time. The patch doesn't do that.
That was in comparison to HEAD, not your patch.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 00:37 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-13 00:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 3:56 PM Andres Freund <andres@anarazel.de> wrote:
> > Why is that simpler than a boolean array, which represents whether or
> > not the item has had its heap_prune_record_unused() call yet (if it's
> > a tuple with storage)?
>
> Well, your change is basically a new approach of pruning - a much better
> one. But it's a larger change than just eliminating the repeated HTSV calls so
> that they cannot change over time. That'd be ~10-15 lines.
Before Wednesday, I had no idea that a HEAPTUPLE_DELETE_IN_PROGRESS
could turn to DEAD during VACUUM/pruning at all. And so I now have to
wonder: what else don't I know?
There is one thing that I am fairly confident of here: the HOT chain
validation stuff is very robust. Experience has shown the invariants
to be very reliable (and if they're not reliable then we're in big
trouble anyway). The invariants can be leveraged to make the pruning
fix robust against both the issue we know about, and hypothetical
undiscovered issues that also may affect pruning. The overall risk
seems much lower with my patch. It isn't the smallest possible patch,
certainly, but that doesn't seem all that relevant, given the
specifics of the situation.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 01:48 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-13 01:48 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On 2021-11-12 16:37:24 -0800, Peter Geoghegan wrote:
> There is one thing that I am fairly confident of here: the HOT chain
> validation stuff is very robust. Experience has shown the invariants
> to be very reliable (and if they're not reliable then we're in big
> trouble anyway).
What experience? Isn't this all new stuff?
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 01:57 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-13 01:57 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 5:48 PM Andres Freund <andres@anarazel.de> wrote:
> On 2021-11-12 16:37:24 -0800, Peter Geoghegan wrote:
> > There is one thing that I am fairly confident of here: the HOT chain
> > validation stuff is very robust. Experience has shown the invariants
> > to be very reliable (and if they're not reliable then we're in big
> > trouble anyway).
>
> What experience? Isn't this all new stuff?
I meant heapam since 2007, when HOT was first added -- all of it.
Notably, v4 of the patch makes the most conservative possible
assumptions about how HTSV might change its mind about an XID -- no
more "A RECENTLY_DEAD tuple is actually sometimes DEAD in the specific
context of processing a HOT chain". Now it's more like "A DEAD tuple
is actually sometimes RECENTLY_DEAD at the level of a HOT chain,
except for disconnected/aborted heap-only tuples, and if you don't
like that (i.e. during VACUUM) you can just retry pruning from
scratch immediately afterwards".
You said it yourself: who knows exactly what the justification for
RECENTLY_DEAD->DEAD was? I have to imagine it had something to do with the
"INSERT_IN_PROGRESS becomes DEAD due to concurrent xact abort" thing,
but that's unclear. And even if it was clear, and even if we knew that
it was 100% safe at one point, it still wouldn't be clear that it's
safe today, in Postgres 14.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 02:32 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-13 02:32 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Nov 12, 2021 at 5:57 PM Peter Geoghegan <pg@bowt.ie> wrote:
> You said it yourself: who knows exactly what the justification for
> RECENTLY_DEAD->DEAD was? I have to imagine it had something to do with the
> "INSERT_IN_PROGRESS becomes DEAD due to concurrent xact abort" thing,
> but that's unclear. And even if it was clear, and even if we knew that
> it was 100% safe at one point, it still wouldn't be clear that it's
> safe today, in Postgres 14.
Another relevant factor is how we deal with already-corrupt HOT chains
affected by the bug. I would be comfortable with a full "can't happen"
error in the new code path for disconnected and aborted heap-only
tuples, provided the error only gets raised when the tuple is fully
LIVE according to HTSV (and also assert that it's DEAD). Something
like my v4 plus this LIVE-should-be-DEAD defensive error seems very
likely to avoid making the corruption any worse. There is a huge
amount of redundancy in the tuple headers that we can cross check
inexpensively.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 15:06 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 2 replies; 90+ messages in thread
From: Dmitry Dolgov @ 2021-11-13 15:06 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Andres Freund <andres@anarazel.de>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On Fri, Nov 12, 2021 at 02:46:22PM -0800, Peter Geoghegan wrote:
> On Fri, Nov 12, 2021 at 2:29 PM Andres Freund <andres@anarazel.de> wrote:
> > > Naturally, I also went through the exercise of trying to find a
> > > counterexample, where pruning doesn't see a disconnected tuple as DEAD
> > > in its HTSV. I could not get the assertion to fail with Alexander's
> > > test case, nor with make check-world.
> >
> > I don't think that provides a meaningful coverage. Alexander's test has a
> > quite limited set operations (which e.g. doesn't include an subxacts), and our
> > own tests around subtransactions, and particularly concurrent subtransaction
> > heavy work, is quite, uh, minimal.
>
> It's a start.
>
> We need to be pragmatic here. There is some uncertainty about what
> HTSV might say about a disconnected tuple in the absence of
> corruption, or there is a risk of a new problem like that coming up in
> the future -- let's work within those confines, then. What do you want
> to do about that? There aren't that many choices, since, to repeat,
> the tuple is "morally" DEAD no matter what. Even with corruption, even
> without corruption in the presence of some unanticipated corner case
> with HTSV -- this is fundamental.
I've got curious if modifying the Alexander's test case could reveal
something interesting, and sprinkled it with savepoints and rollbacks.
Almost immediately a new problem has manifested itself, although the
crash has nothing to do with the disconnected tuples as far as I can
tell -- still probably worth mentioning. In this case vacuum invoked
lazy_scan_prune, and during the first scan one of the chains had a
HEAPTUPLE_DEAD at the third position. The processing flow fell through
to heap_prune_record_prunable and crashed on an assert with an
InvalidTransactionId:
#3 0x000055a2b260d1f9 in heap_prune_record_prunable (prstate=0x7ffd0c0ecdf0, xid=0) at pruneheap.c:872
#4 0x000055a2b260ca72 in heap_prune_chain (buffer=2117, rootoffnum=150, prstate=0x7ffd0c0ecdf0) at pruneheap.c:695
#5 0x000055a2b260bcd6 in heap_page_prune (relation=0x7fb98e217e20, buffer=2117, vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, old_snap_xmin=0, old_snap_ts=0, report_stats=false, off_loc=0x55a2b3e6a0cc) at pruneheap.c:288
#6 0x000055a2b261309c in lazy_scan_prune (vacrel=0x55a2b3e6a060, buf=2117, blkno=192, page=0x7fb97856bf80 "", vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, prunestate=0x7ffd0c0ee9d0) at vacuumlazy.c:1739
Applying heap_prune_record_prunable only if TransactionIdIsNormal seems
to help. The original implementation didn't reach
heap_prune_record_prunable either and also doesn't crash.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-13 17:06 Peter Geoghegan <pg@bowt.ie>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
1 sibling, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-13 17:06 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Nov 13, 2021 at 7:05 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
> I've got curious if modifying the Alexander's test case could reveal
> something interesting, and sprinkled it with savepoints and rollbacks.
> Almost immediately a new problem has manifested itself, although the
> crash has nothing to do with the disconnected tuples as far as I can
> tell -- still probably worth mentioning. In this case vacuum invoked
> lazy_scan_prune, and during the first scan one of the chains had a
> HEAPTUPLE_DEAD at the third position. The processing flow fell through
> to heap_prune_record_prunable and crashed on an assert with an
> InvalidTransactionId:
Is this just with the bugfix applied? I think that it is. Looks like a
minor bug to me.
I think that I need to consistently "break" in the DEAD case, to avoid
ending up here. In other words, it should not literally be
"reinterpreted" as RECENTLY_DEAD by falling through in the switch
statement (though the concept of reinterpreting certain DEAD tuples as
RECENTLY_DEAD still seems perfectly sound).
Here's why the assertion (invalid xmax/update xid cannot be used in
heap_prune_record_prunable() call) fails:
DEAD means that you might not have a valid update XID -- aborted
update is what we expect. But RECENTLY_DEAD means that there must have
been a deleter xact, and that the xact must have committed (can't have
been that the inserter aborted). This is a consequence of the fact
that the tuple is at least still visible to somebody (or could be),
unlike in the DEAD case. And so xmax must be a valid XID, and so the
existing RECENTLY_DEAD case handling can legitimately always expect
that. But I cannot (and should not) allow a call to
heap_prune_record_prunable() with a DEAD-to-HTSV tuple, even when I
"reinterpret" it as RECENTLY_DEAD in order to make a clean
determination of the tuple to delete up until for the entire HOT
chain.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-16 02:39 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-16 02:39 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-10 19:08:21 -0800, Andres Freund wrote:
> On 2021-11-09 19:07:02 -0800, Peter Geoghegan wrote:
> > @@ -700,11 +720,18 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
> > }
> >
> > /*
> > - * Remember the last DEAD tuple seen. We will advance past
> > - * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
> > - * but we can't advance past anything else. We have to make sure that
> > - * we don't miss any DEAD tuples, since DEAD tuples that still have
> > - * tuple storage after pruning will confuse VACUUM.
> > + * Remember the last DEAD tuple seen from this HOT chain.
> > + *
> > + * We expect to sometimes find a RECENTLY_DEAD tuple after a DEAD
> > + * tuple. When this happens, the RECENTLY_DEAD tuple will be treated
> > + * as if it was DEAD all along. Manage that now.
>
> I now actually wonder why this is correct. There's another comment about it:
>
> > We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
> > * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
>
> But that doesn't justify very much.
>
> What prevents the scenario that some other backend e.g. has a snapshot with
> xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
> than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
> row, but *not* the RECENTLY_DEAD one.
I think the code is actually correct if awfully commented. I.e. the above
scenario cannot happen in a harmful situation. The relevant piece is that the
set of "valid" snapshots is limited (for a specific cluster history).
A row can only be RECENTLY_DEAD if the deleting / updating transaction
committed.
An intermediary RECENTLY_DEAD row followed by a DEAD row would only be visible
to a snapshot that either had
1) row.xmax >= Snapshot.xmax
2) row.xmax in Snapshot.xip.
There cannot be a snapshot that sees the RECENTLY_DEAD row while also seeing
the subsequent row as DEAD, because the existence of one of the two conditions
would have held back the xid horizon:
1) is not possible, because Snapshot.xmin is <= Snapshot.xmax, and
Snapshot.xmin >= MyProc->xmin. The subsequent row could not be DEAD.
2) is not possible, because .xip only contains xids >= Snapshot.xmin.
IOW, there *can* be a snapshot with xmin=xmax=RECENTLY_DEAD-row, but it'd not
see the RECENTLY_DEAD row anyway.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-17 00:22 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-17 00:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Nov 15, 2021 at 6:39 PM Andres Freund <andres@anarazel.de> wrote:
> > What prevents the scenario that some other backend e.g. has a snapshot with
> > xmin=xmax=RECENTLY_DEAD-row. If the RECENTLY_DEAD row has an xid that is later
> > than the DEAD row, this afaict would make it perfectly legal to prune the DEAD
> > row, but *not* the RECENTLY_DEAD one.
>
> I think the code is actually correct if awfully commented. I.e. the above
> scenario cannot happen in a harmful situation. The relevant piece is that the
> set of "valid" snapshots is limited (for a specific cluster history).
> IOW, there *can* be a snapshot with xmin=xmax=RECENTLY_DEAD-row, but it'd not
> see the RECENTLY_DEAD row anyway.
I'm not surprised that this turned out to be correct. It hasn't made
me change my mind; I still believe that the best way forward is to
backpatch a more comprehensive fix, like my patch. Again, I just think
that that approach has the best chance of avoiding any further
problems.
The alternative proposal that you lean towards (minimal fix on 14,
commit the comprehensive fix to HEAD only) seems fine overall, though.
In any case I don't think that there is much point in further debating
it. If I was going to convince you, it would have happened by now. It
is your bug (kind of), and so I defer to you on this.
Why don't you produce a minimal fix for backpatch? I'll review that,
just as you reviewed my patch.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-18 07:19 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-18 07:19 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
I spent quite a bit more time trying to have a good testcase for reproducing
this reliably. I didn't feel confident using testing that takes substantial
amounts of time to hit bugs.
On 2021-11-10 11:20:10 -0800, Andres Freund wrote:
> The way this definitely breaks - I have been able to reproduce this in
> isolation - is when one tuple is processed twice by heap_prune_chain(), and
> the result of HeapTupleSatisfiesVacuum() changes from
> HEAPTUPLE_DELETE_IN_PROGRESS to DEAD.
>
> Consider a page like this:
>
> lp 1: redirect to lp2
> lp 2: deleted by xid x, not yet committed
>
> and a sequence of events like this:
>
> 1) heap_prune_chain(rootlp = 1)
> 2) commit x
> 3) heap_prune_chain(rootlp = 2)
>
> 1) heap_prune_chain(rootlp = 1) will go to lp2, and see a
> HEAPTUPLE_DELETE_IN_PROGRESS and thus not do anything.
>
> 3) then could, with the snapshot scalability changes, get DEAD back from
> HTSV. Due to the "fuzzy" nature of the post-snapshot-scalability xid horizons,
> that is possible, because we can end up rechecking the boundary condition and
> seeing that now the horizon allows us to prune x / lp2.
>
> At that point we have a redirect tuple pointing into an unused slot. Which is
> "illegal", because something independent can be inserted into that slot.
This above isn't quite right - it turns out this requires additional steps to
be triggerable (without debugging code to make things easier to hit). As shown
above the result won't change, because the xmin horizon of x will have held
the xmin horizon far enough back that GlobalVisTestShouldUpdate() won't ever
recompute the horizon for x on its own.
However, what *can* happen is that *another* tuple is within the "fuzzy"
horizon window, and that tuple can trigger a recomputation of the
horizon. Which then basically triggers the above problem.
However even that is very hard to hit, because normally we won't recompute the
horizon in that case either, because of
* The current heuristic is that we update only if RecentXmin has changed
* since the last update. If the oldest currently running transaction has not
* finished, it is unlikely that recomputing the horizon would be useful.
I.e. a snapshot needs to be computed between the the vacuum_set_xid_limit()
and the heap_page_prune() call. Turns out, there's a window:
vac_open_indexes() needs to lock the indexes. Which in turn will trigger
AcceptInvalidationMessages(). Relcache invalidations then can trigger a
catalog snapshot to be built, which can increase RecentXmin, and later trigger
a new accurate horizon to be built.
Consider a page like:
lp 1: redirect to lp 3
lp 2: RECENTLY_DEAD tuple xmax x1, held back by session y's xmin horizon
lp 3: DELETE_IN_PROGRESS tuple xmax x2
and the following sequence of actions:
1) vacuum_set_xid_limit()
-> sets ComputeXidHorizonsResultLastXmin = RecentXmin
2) session y commits
3) vac_open_indexes()
-> AcceptInvalidationMessages()
-> InvalidateCatalogSnapshot()
-> RelationCacheInvalidateEntry()
-> GetCatalogSnapshot()
-> GetSnapshotData()
updates RecentXmin
4) x2 commits
5) heap_prune_chain(1)
-> sees lp3 as RECENTLY_DEAD (or DELETE_IN_PROGRESS depending on timing)
-> doesn't do anything
6) heap_prune_chain(2)
-> sees a RECENTLY_DEAD, updates GlobalVisTestShouldUpdate() says yes,
updates xid horizon to above x2
7) heap_prune_chain(3)
-> uses the HeapOnly path at the beginning, sees a DEAD tuple, marks the
element as unused
Boom.
Not sure if it's worth it, but I think this can be made into a reliably
isolationtest by causing a tables index to locked exclusively, blocking
vac_open_indexes(), which can then reliably schedule a new relcache inval,
which in turn can compute a new RecentXmin. Might be too fragile to be worth
it.
As an independent issue, I don't think it can ever be safe to do catalog
access with PROC_IN_VACUUM set. Which vac_open_indexes() (and likely some
other things) clearly do. Without the current backend's xmin set, another
backend can remove still required tuples. Which can cause us to build corrupt
relcache entries, and then subsequently skip indexing some indexes, causing
corruption. I think I hit this while trying to repro the above, but it's hard
to be sure.
> I hit a crash once in 13 with a slightly evolved version of the test (many
> connections creating / dropping the partitions as in the original scenario,
> using :client_id to target different tables). It's possible that my
> instrumentation was the cause of that. Unfortunately it took quite a few hours
> to hit the problem in 13...
I wonder if this was actually the above PROC_IN_VACUUM thing.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-19 00:36 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-19 00:36 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-17 23:19:41 -0800, Andres Freund wrote:
> Not sure if it's worth it, but I think this can be made into a reliably
> isolationtest by causing a tables index to locked exclusively, blocking
> vac_open_indexes(), which can then reliably schedule a new relcache inval,
> which in turn can compute a new RecentXmin. Might be too fragile to be worth
> it.
Attached is such an isolationtest. In an unmodified HEAD it ends up with
step s1_select_1:
SET LOCAL enable_seqscan = false;
SELECT ctid, /*xmin, xmax, */ id FROM many_updates WHERE id = 1;
RESET enable_seqscan;
ctid |id
-----+--
(0,3)|17
(1 row)
without amcheck detecting corruption at that point:(.
It's clearly not yet something we could consider committable, but I think it
might be a useful basis for such a test.
Greetings,
Andres Freund
Attachments:
v1-0001-wip-test-heap_prune_page-bug-recently-dead-dead-i.patch (text/x-diff, 0B) download
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-19 02:55 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-19 02:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Mark Dilger <mark.dilger@enterprisedb.com>
On Thu, Nov 18, 2021 at 4:36 PM Andres Freund <andres@anarazel.de> wrote:
> Attached is such an isolationtest. In an unmodified HEAD it ends up with
> without amcheck detecting corruption at that point:(.
The best way to teach amcheck to detect this kind of thing is bound to
be verification of HOT chains themselves:
https://www.postgresql.org/message-id/flat/CAH2-Wznphpd490o%2Bur_bi6-YmC47hRu3Qyxod72RLqC-Uvuhcg%40m...
It would also be nice to teach verify_nbtree.c to look out for the
presence of index tuples that shouldn't be there during heapallindexed
verification, but that's significantly harder.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-19 18:55 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Andres Freund @ 2021-11-19 18:55 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org, Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>; Mark Dilger <mark.dilger@enterprisedb.com>
Hi,
On November 18, 2021 6:55:08 PM PST, Peter Geoghegan <pg@bowt.ie> wrote:
>On Thu, Nov 18, 2021 at 4:36 PM Andres Freund <andres@anarazel.de> wrote:
>> Attached is such an isolationtest. In an unmodified HEAD it ends up with
>
>> without amcheck detecting corruption at that point:(.
>
>The best way to teach amcheck to detect this kind of thing is bound to
>be verification of HOT chains themselves:
>
>https://www.postgresql.org/message-id/flat/CAH2-Wznphpd490o%2Bur_bi6-YmC47hRu3Qyxod72RLqC-Uvuhcg%40m...
That'd not reliably catch this kind of thing, because the hot chains easily can end up correct enough looking.
>It would also be nice to teach verify_nbtree.c to look out for the
>presence of index tuples that shouldn't be there during heapallindexed
>verification, but that's significantly harder.
I think what we really need is a pass that verifies that heap tuples reached via the index actually match the index key. There's just too many things that can break that. It'll not be blazingly fast, but there's just way too many things that other approaches just won't find.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-22 17:59 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-22 17:59 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-16 16:22:06 -0800, Peter Geoghegan wrote:
> I'm not surprised that this turned out to be correct. It hasn't made
> me change my mind; I still believe that the best way forward is to
> backpatch a more comprehensive fix, like my patch. Again, I just think
> that that approach has the best chance of avoiding any further
> problems.
>
> The alternative proposal that you lean towards (minimal fix on 14,
> commit the comprehensive fix to HEAD only) seems fine overall, though.
> In any case I don't think that there is much point in further debating
> it. If I was going to convince you, it would have happened by now. It
> is your bug (kind of), and so I defer to you on this.
I'm not quite sure what you referring to with "convincing me"? Whether to go
for something comprehensive or more minimal? I'm kind of agnostic on what the
right approach is.
> Why don't you produce a minimal fix for backpatch? I'll review that,
> just as you reviewed my patch.
Here's a draft of that change.
I think it's worth to clean up the regression test I wrote and use it
regardless of which version of the fix we end up choosing. But I'm a bit bit
on the fence - it's quite complicated.
OTOH, I think with the framework in place it'd not be too hard to write a few
more tests for odd pruning scenarios...
If we were to, it'd probably best to to comment out the mon_* stuff, as that
is what requires the amcheck / pageinspect extensions, which we'd likely not
want to depend on. But it's pretty crucial for development.
Greetings,
Andres Freund
Attachments:
v3-0001-Test-heap_prune_page-bug-w-RECENTLY_DEAD-DEAD-acr.patch (text/x-diff, 0B) download
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-22 20:41 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-11-22 20:41 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Mon, Nov 22, 2021 at 9:59 AM Andres Freund <andres@anarazel.de> wrote:
> I'm not quite sure what you referring to with "convincing me"? Whether to go
> for something comprehensive or more minimal? I'm kind of agnostic on what the
> right approach is.
I thought that you were clearly leaning in the direction of a
minimal-only fix for backpatch. I just meant that there is no point in
saying much more about it. It's not like I have a problem with your
approach. My personal opinion is that we'd be somewhat better off if
we went with my more comprehensive approach, even on 14. That's not
how you see it - which is fair enough (reasoning about these risks is
hard, and it tends to be quite subjective).
And so, as I said before, I can leave it at that.
> > Why don't you produce a minimal fix for backpatch? I'll review that,
> > just as you reviewed my patch.
>
> Here's a draft of that change.
Looks reasonable to me.
> I think it's worth to clean up the regression test I wrote and use it
> regardless of which version of the fix we end up choosing. But I'm a bit bit
> on the fence - it's quite complicated.
+1 to the idea of keeping it somewhere. Without necessarily running it
on the BF.
> OTOH, I think with the framework in place it'd not be too hard to write a few
> more tests for odd pruning scenarios...
Can we commit a test like this without running it by default, at all?
It's not like it has never been done before.
> If we were to, it'd probably best to to comment out the mon_* stuff, as that
> is what requires the amcheck / pageinspect extensions, which we'd likely not
> want to depend on. But it's pretty crucial for development.
Good idea.
The nice thing about using amcheck for something like this (compared
to pageinspect) is that we don't have to specify exact details about
what it looks like when the test passes. Because passing the test just
means the absence of any error.
How hard would it be to just add amcheck coverage on HEAD? Something
that goes just a bit further with LP_REDIRECT validation (currently
verify_heapam does practically nothing like that). We should add
comprehensive HOT chain validation too, but that can wait a little
while longer. It's a lot more complicated.
Another thing is that the new assertions themselves are part of our
overall testing strategy. I see that you've gone further with that
than I did in your v3-0003-* patch. I should adopt that code to my
more comprehensive fix for HEAD, while still keeping around the
existing heap_prune_chain() assertions (perhaps just as
documentation/comments to make the code easier to follow).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-11-22 22:34 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-11-22 22:34 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-22 12:41:32 -0800, Peter Geoghegan wrote:
> On Mon, Nov 22, 2021 at 9:59 AM Andres Freund <andres@anarazel.de> wrote:
> > I'm not quite sure what you referring to with "convincing me"? Whether to go
> > for something comprehensive or more minimal? I'm kind of agnostic on what the
> > right approach is.
>
> I thought that you were clearly leaning in the direction of a
> minimal-only fix for backpatch. I just meant that there is no point in
> saying much more about it. It's not like I have a problem with your
> approach. My personal opinion is that we'd be somewhat better off if
> we went with my more comprehensive approach, even on 14. That's not
> how you see it - which is fair enough (reasoning about these risks is
> hard, and it tends to be quite subjective).
I think I'm actually just not sure what the better approach for the
backbranches is. I'm worried about the size of your patch, I'm worried about
the craziness of the current architecture (if you can call it that) of
heap_page_prune() with my patch.
I think either way your approach should incorporate the dedicated loop to
check the visibility - for performance reasons alone.
> > I think it's worth to clean up the regression test I wrote and use it
> > regardless of which version of the fix we end up choosing. But I'm a bit bit
> > on the fence - it's quite complicated.
>
> +1 to the idea of keeping it somewhere. Without necessarily running it
> on the BF.
>
> > OTOH, I think with the framework in place it'd not be too hard to write a few
> > more tests for odd pruning scenarios...
>
> Can we commit a test like this without running it by default, at all?
> It's not like it has never been done before.
Is there a reason not to run it if we commit it? IME tests not run by default
tend to bitrot very quickly. The only issue that I can think of is that it
might be hard to make the test useful and robust in the presence of
debug_discard_caches != 0.
> The nice thing about using amcheck for something like this (compared
> to pageinspect) is that we don't have to specify exact details about
> what it looks like when the test passes. Because passing the test just
> means the absence of any error.
Unless amcheck gets more in-depth there's just too much it doesn't show :(
> How hard would it be to just add amcheck coverage on HEAD? Something
> that goes just a bit further with LP_REDIRECT validation (currently
> verify_heapam does practically nothing like that). We should add
> comprehensive HOT chain validation too, but that can wait a little
> while longer. It's a lot more complicated.
What exactly do you mean with "amcheck coverage" in this context?
> Another thing is that the new assertions themselves are part of our
> overall testing strategy. I see that you've gone further with that
> than I did in your v3-0003-* patch. I should adopt that code to my
> more comprehensive fix for HEAD, while still keeping around the
> existing heap_prune_chain() assertions (perhaps just as
> documentation/comments to make the code easier to follow).
I found that style of check helpful because it notices problems sooner. What
took the longest debugging this issue was that simple assertions (e.g. during
heap_page_prune_execute() or heap_prune_record_unused()) would trigger well
after the actual bug. E.g. marking an item unused that is referenced by a
pre-existing LP_REDIRECT item can't easily be detected at that time. It'd
trigger an assertion at the time the LP_REDIRECT item is marked dead, but that
doesn't help to pinpoint the origin of the problem much.
I'm inclined to think that the costs of the check are worth incurring by
default for cassert builds.
I wish I knew of a good way to have pgbench check the validity of its
results. There's a huge difference between not crashing and returning correct
results. And there's just a lot of bugs that will result in valid-enough
on-disk/in-buffer data, but which are not correct.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-11 04:57 Andres Freund <andres@anarazel.de>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 90+ messages in thread
From: Andres Freund @ 2021-12-11 04:57 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-22 14:34:36 -0800, Andres Freund wrote:
> I think I'm actually just not sure what the better approach for the
> backbranches is. I'm worried about the size of your patch, I'm worried about
> the craziness of the current architecture (if you can call it that) of
> heap_page_prune() with my patch.
Given the lack of further discussion, let's go with the "minimal" fix and your
stuff later. Attached is further polished patch - no significant changes,
plenty copy-editing stuff.
0001 is the bugfix
0002 adds the additional assertions - I'm wondering about only committing that
in HEAD?
I think your patch should easily be able to use prstate->htsv?
0003 removes the redundant RelationGetNumberOfBlocks() calls. As 0001 does not
change the total number of RelationGetNumberOfBlocks() calls I'm inclined to
just apply this to HEAD?
0004 is a patch that tries to address your point about the GlobalVisTestFor()
placement, sort of
My earlier point that moving it to earlier would be a bad idea because it'd
make the horizon "older" was bogus - GlobalVisTestFor() doesn't itself do
any horizon determination. However moving the GlobalVisTestFor() earlier
still seems wrong - imo the vacuum_set_xid_limits() call should be moved to
later. The attached patch moves both, wrapped in a new function, to just
before the scan in lazy_scan_heap().
This clearly is HEAD only material - I'm only bringing it up here because
the issue of the GlobalVisTestFor() placement was raised in here...
> > > I think it's worth to clean up the regression test I wrote and use it
> > > regardless of which version of the fix we end up choosing. But I'm a bit bit
> > > on the fence - it's quite complicated.
> >
> > +1 to the idea of keeping it somewhere. Without necessarily running it
> > on the BF.
> >
> > > OTOH, I think with the framework in place it'd not be too hard to write a few
> > > more tests for odd pruning scenarios...
> >
> > Can we commit a test like this without running it by default, at all?
> > It's not like it has never been done before.
>
> Is there a reason not to run it if we commit it? IME tests not run by default
> tend to bitrot very quickly. The only issue that I can think of is that it
> might be hard to make the test useful and robust in the presence of
> debug_discard_caches != 0.
I tried pretty hard to get the test to be reliable enough to commit it. But in
the end I think using the index locking as a "control" mechanism just isn't
great - as evidenced by 0003 breaking the test entirely. I think unfortunately
there's not much hope for testing this unless we get wait points - which seems
not too close :(.
Greetings,
Andres Freund
Attachments:
v4-0001-Fix-possible-HOT-corruption-when-RECENTLY_DEAD-ch.patch (text/x-diff, 0B) download
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-11 04:58 Andres Freund <andres@anarazel.de>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2021-12-11 04:58 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2021-11-13 16:06:40 +0100, Dmitry Dolgov wrote:
> I've got curious if modifying the Alexander's test case could reveal
> something interesting, and sprinkled it with savepoints and rollbacks.
> Almost immediately a new problem has manifested itself, although the
> crash has nothing to do with the disconnected tuples as far as I can
> tell -- still probably worth mentioning. In this case vacuum invoked
> lazy_scan_prune, and during the first scan one of the chains had a
> HEAPTUPLE_DEAD at the third position. The processing flow fell through
> to heap_prune_record_prunable and crashed on an assert with an
> InvalidTransactionId:
>
> #3 0x000055a2b260d1f9 in heap_prune_record_prunable (prstate=0x7ffd0c0ecdf0, xid=0) at pruneheap.c:872
> #4 0x000055a2b260ca72 in heap_prune_chain (buffer=2117, rootoffnum=150, prstate=0x7ffd0c0ecdf0) at pruneheap.c:695
> #5 0x000055a2b260bcd6 in heap_page_prune (relation=0x7fb98e217e20, buffer=2117, vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, old_snap_xmin=0, old_snap_ts=0, report_stats=false, off_loc=0x55a2b3e6a0cc) at pruneheap.c:288
> #6 0x000055a2b261309c in lazy_scan_prune (vacrel=0x55a2b3e6a060, buf=2117, blkno=192, page=0x7fb97856bf80 "", vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, prunestate=0x7ffd0c0ee9d0) at vacuumlazy.c:1739
>
> Applying heap_prune_record_prunable only if TransactionIdIsNormal seems
> to help. The original implementation didn't reach
> heap_prune_record_prunable either and also doesn't crash.
Does your modified test still find problems with 0001 & 0002 from
https://postgr.es/m/20211211045710.ljtuu4gfloh754rs%40alap3.anarazel.de
applied?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-11 22:03 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2021-12-11 22:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Dec 10, 2021 at 8:57 PM Andres Freund <andres@anarazel.de> wrote:
> Given the lack of further discussion, let's go with the "minimal" fix and your
> stuff later. Attached is further polished patch - no significant changes,
> plenty copy-editing stuff.
Makes sense.
> 0001 is the bugfix
LGTM.
> 0002 adds the additional assertions - I'm wondering about only committing that
> in HEAD?
I don't think that it makes much difference, since these are only
assertions. I probably wouldn't bother with that myself, but I also
have zero concern about it breaking anything.
> I think your patch should easily be able to use prstate->htsv?
Let's get this committed, to 14 and HEAD. And then leave it to me to
rebase, as additional HEAD-only hardening.
> 0003 removes the redundant RelationGetNumberOfBlocks() calls. As 0001 does not
> change the total number of RelationGetNumberOfBlocks() calls I'm inclined to
> just apply this to HEAD?
Yeah, that can be part of my follow-up hardening patch, which (to
repeat) will be HEAD-only.
> 0004 is a patch that tries to address your point about the GlobalVisTestFor()
> placement, sort of
>
> My earlier point that moving it to earlier would be a bad idea because it'd
> make the horizon "older" was bogus - GlobalVisTestFor() doesn't itself do
> any horizon determination. However moving the GlobalVisTestFor() earlier
> still seems wrong - imo the vacuum_set_xid_limits() call should be moved to
> later. The attached patch moves both, wrapped in a new function, to just
> before the scan in lazy_scan_heap().
I think that this is the right direction, and I like the comments a
lot. But I don't like the idea of putting this initialization into
lazy_scan_heap(), for purely structural reasons. I plan to get rid of
the VacuumParams argument to lazy_scan_heap() completely, which would
make "VacuumParams setup" the sole responsibility of its caller,
heap_vacuum_rel(). The draft version of my "remove special cases to
advance relfrozenxid" patch (the one you've reviewed) already does
this. In general I really think it's important to make the state in
vacuumlazy.c as simple as possible.
I have no objection to delaying the lazy_scan_heap_limits() stuff
until right before lazy_scan_heap() is called. However, I do think
that we should always know certain basic "immutable" facts about a
VACUUM at the point that we call lazy_scan_heap(), which is not the
case with this patch.
Honestly, I'm surprised that you see much value in delaying the
lazy_scan_heap_limits() stuff until the very last microsecond. How
many microseconds could we possibly delay it by?
> I tried pretty hard to get the test to be reliable enough to commit it. But in
> the end I think using the index locking as a "control" mechanism just isn't
> great - as evidenced by 0003 breaking the test entirely. I think unfortunately
> there's not much hope for testing this unless we get wait points - which seems
> not too close :(.
Hopefully somebody will seriously commit to that at some point. It's
important, but not very glamorous.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-11 22:51 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2021-12-11 22:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Dec 11, 2021 at 2:03 PM Peter Geoghegan <pg@bowt.ie> wrote:
> I have no objection to delaying the lazy_scan_heap_limits() stuff
> until right before lazy_scan_heap() is called. However, I do think
> that we should always know certain basic "immutable" facts about a
> VACUUM at the point that we call lazy_scan_heap(), which is not the
> case with this patch.
>
> Honestly, I'm surprised that you see much value in delaying the
> lazy_scan_heap_limits() stuff until the very last microsecond. How
> many microseconds could we possibly delay it by?
Have you thought about the implications for the ongoing work to set
pg_class.relfrozenxid to the oldest observed XID in the table, instead
of just using FreezeLimit naively (which I've prototyped but haven't
posted)?
What if the target heap relation gets extended after we've established
nblocks/rel_pages for the lazyvacuum.c operation (by calling
RelationGetNumberOfBlocks()), but before we get to the
lazy_scan_heap_limits() stuff for the same operation? What if there
are a small number of heap pages at the end of the relation that we
won't get to at all in the ongoing VACUUM? They could have heap tuples
whose header XIDs are from just before our OldestXmin cutoff. I
believe it follows that we cannot miss them (at least not in an
aggressive VACUUM, maybe not ever with my patch).
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-13 12:21 Dmitry Dolgov <9erthalion6@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Dmitry Dolgov @ 2021-12-13 12:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
> On Fri, Dec 10, 2021 at 08:58:26PM -0800, Andres Freund wrote:
> Hi,
>
> On 2021-11-13 16:06:40 +0100, Dmitry Dolgov wrote:
> > I've got curious if modifying the Alexander's test case could reveal
> > something interesting, and sprinkled it with savepoints and rollbacks.
> > Almost immediately a new problem has manifested itself, although the
> > crash has nothing to do with the disconnected tuples as far as I can
> > tell -- still probably worth mentioning. In this case vacuum invoked
> > lazy_scan_prune, and during the first scan one of the chains had a
> > HEAPTUPLE_DEAD at the third position. The processing flow fell through
> > to heap_prune_record_prunable and crashed on an assert with an
> > InvalidTransactionId:
> >
> > #3 0x000055a2b260d1f9 in heap_prune_record_prunable (prstate=0x7ffd0c0ecdf0, xid=0) at pruneheap.c:872
> > #4 0x000055a2b260ca72 in heap_prune_chain (buffer=2117, rootoffnum=150, prstate=0x7ffd0c0ecdf0) at pruneheap.c:695
> > #5 0x000055a2b260bcd6 in heap_page_prune (relation=0x7fb98e217e20, buffer=2117, vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, old_snap_xmin=0, old_snap_ts=0, report_stats=false, off_loc=0x55a2b3e6a0cc) at pruneheap.c:288
> > #6 0x000055a2b261309c in lazy_scan_prune (vacrel=0x55a2b3e6a060, buf=2117, blkno=192, page=0x7fb97856bf80 "", vistest=0x55a2b31d2d60 <GlobalVisCatalogRels>, prunestate=0x7ffd0c0ee9d0) at vacuumlazy.c:1739
> >
> > Applying heap_prune_record_prunable only if TransactionIdIsNormal seems
> > to help. The original implementation didn't reach
> > heap_prune_record_prunable either and also doesn't crash.
>
> Does your modified test still find problems with 0001 & 0002 from
> https://postgr.es/m/20211211045710.ljtuu4gfloh754rs%40alap3.anarazel.de
> applied?
Nope, everything seems to be working smoothly.
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2021-12-13 18:48 Andres Freund <andres@anarazel.de>
parent: Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 90+ messages in thread
From: Andres Freund @ 2021-12-13 18:48 UTC (permalink / raw)
To: Dmitry Dolgov <9erthalion6@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On 2021-12-13 13:21:54 +0100, Dmitry Dolgov wrote:
> > On Fri, Dec 10, 2021 at 08:58:26PM -0800, Andres Freund wrote:
> > Does your modified test still find problems with 0001 & 0002 from
> > https://postgr.es/m/20211211045710.ljtuu4gfloh754rs%40alap3.anarazel.de
> > applied?
>
> Nope, everything seems to be working smoothly.
Thanks!
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-10 19:31 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-01-10 19:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Dec 10, 2021 at 8:57 PM Andres Freund <andres@anarazel.de> wrote:
> Given the lack of further discussion, let's go with the "minimal" fix and your
> stuff later. Attached is further polished patch - no significant changes,
> plenty copy-editing stuff.
We have roughly 3 weeks to get this into 14.2. Can you push your
minimal fix soon? Any blockers?
Thanks
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-12 19:25 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-01-12 19:25 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-01-10 11:31:58 -0800, Peter Geoghegan wrote:
> On Fri, Dec 10, 2021 at 8:57 PM Andres Freund <andres@anarazel.de> wrote:
> > Given the lack of further discussion, let's go with the "minimal" fix and your
> > stuff later. Attached is further polished patch - no significant changes,
> > plenty copy-editing stuff.
>
> We have roughly 3 weeks to get this into 14.2. Can you push your
> minimal fix soon?
Working on it now. I've a meeting or two in a bit, it'll probably be after...
> Any blockers?
I'm just struggling with / procrastinating on the commit message, tbh. The
whole issue is kinda complicated to explain... :/
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-12 21:05 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-01-12 21:05 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Wed, Jan 12, 2022 at 11:25 AM Andres Freund <andres@anarazel.de> wrote:
> > Any blockers?
>
> I'm just struggling with / procrastinating on the commit message, tbh. The
> whole issue is kinda complicated to explain... :/
I think that it would make sense for the commit message to frame the
problem as: pruneheap.c doesn't take sufficient care when traversing
HOT chains to determine their full extent, for the purposes of
pruning. There was a general lack of robustness, and the snapshot
scalability work happened to run into that, resulting in hot chain
corruption under very specific conditions.
If I was in your position I think I would resist framing the problem
in this way; I'd probably be concerned that it would come off as
shifting the blame elsewhere. This high level explanation of things
makes the most sense to me, though. Surely that's the most important
thing.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-14 02:15 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Andres Freund @ 2022-01-14 02:15 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On 2022-01-12 13:05:45 -0800, Peter Geoghegan wrote:
> On Wed, Jan 12, 2022 at 11:25 AM Andres Freund <andres@anarazel.de> wrote:
> > > Any blockers?
> >
> > I'm just struggling with / procrastinating on the commit message, tbh. The
> > whole issue is kinda complicated to explain... :/
After struggling some more, I *finally* pushed the fix and the new assertions.
Thanks for the bug report, investigation, review, etc!
> I think that it would make sense for the commit message to frame the
> problem as: pruneheap.c doesn't take sufficient care when traversing
> HOT chains to determine their full extent, for the purposes of
> pruning. There was a general lack of robustness, and the snapshot
> scalability work happened to run into that, resulting in hot chain
> corruption under very specific conditions.
>
> If I was in your position I think I would resist framing the problem
> in this way; I'd probably be concerned that it would come off as
> shifting the blame elsewhere. This high level explanation of things
> makes the most sense to me, though. Surely that's the most important
> thing.
Thanks!
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-14 17:00 Alexander Lakhin <exclusion@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Alexander Lakhin @ 2022-01-14 17:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
14.01.2022 05:15, Andres Freund wrote:
> On 2022-01-12 13:05:45 -0800, Peter Geoghegan wrote:
>> On Wed, Jan 12, 2022 at 11:25 AM Andres Freund <andres@anarazel.de> wrote:
>>>> Any blockers?
>>> I'm just struggling with / procrastinating on the commit message, tbh. The
>>> whole issue is kinda complicated to explain... :/
> After struggling some more, I *finally* pushed the fix and the new assertions.
>
>
> Thanks for the bug report, investigation, review, etc!
Thank you, Andres and Peter, for the fix!
It was a tough bug to crack for me.
P. S. I'm not seeing the fix in REL_14_STABLE yet, but as soon as it
will be committed I'm going to continue the concurrent installcheck testing.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-01-14 19:00 Andres Freund <andres@anarazel.de>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 0 replies; 90+ messages in thread
From: Andres Freund @ 2022-01-14 19:00 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Peter Geoghegan <pg@bowt.ie>; Dmitry Dolgov <9erthalion6@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On 2022-01-14 20:00:00 +0300, Alexander Lakhin wrote:
> P. S. I'm not seeing the fix in REL_14_STABLE yet, but as soon as it
> will be committed I'm going to continue the concurrent installcheck testing.
Gah, I guess I was too tired last night. Pushed now. Thanks for noticing...
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-02-03 20:54 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Tom Lane @ 2022-02-03 20:54 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Andres Freund <andres@anarazel.de> writes:
>>> I'm just struggling with / procrastinating on the commit message, tbh. The
>>> whole issue is kinda complicated to explain... :/
> After struggling some more, I *finally* pushed the fix and the new assertions.
I'm writing release notes and wondering what I can tell users about
how to detect or recover from this bug. Is a REINDEX sufficient,
or is the presence of the bogus redirect item going to cause
persistent problems? If the latter, can we provide some amelioration?
"Your data is broken and there is nothing you can do about it"
is not nice to be writing in release notes.
regards, tom lane
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-02-04 04:19 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-02-04 04:19 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Peter Geoghegan <pg@bowt.ie>; Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-02-03 15:54:28 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> >>> I'm just struggling with / procrastinating on the commit message, tbh. The
> >>> whole issue is kinda complicated to explain... :/
>
> > After struggling some more, I *finally* pushed the fix and the new assertions.
>
> I'm writing release notes and wondering what I can tell users about
> how to detect or recover from this bug. Is a REINDEX sufficient,
> or is the presence of the bogus redirect item going to cause
> persistent problems?
Good questions.
It's hard to answer whether there's any danger after a REINDEX. Afaics the
build scan would just pick the "lower offset" version of the root
pointer. Which should be fine.
It's possible there could be trouble down the line, e.g. heap pruning doing
something weird once starting in a corrupted state, that then leads REINDEX to
do something bogus. The simple cases look OK, because a second visit/action by
heap_prune_chain for one tid from two different root pointers would see
->marked[offnum] as true. It gets more complicated once multiple intermediary
row versions are involved, because the intermediary row versions won't be in
->marked if an entire chain is pruned. But afaict that should still end up
looking like a hot chain ending in an aborted tuple or such.
> If the latter, can we provide some amelioration?
> "Your data is broken and there is nothing you can do about it"
> is not nice to be writing in release notes.
We've had quite a few bugs around broken HOT chains (which this is an instance
of), I don't think we've provided a useful recipe for any of them :(. We now
have heap amcheck, but it unfortunately doesn't detect most HOT corruption
(redirect pointing to unused is the only case I think).
One ameliorating factor is that, as far as I can tell, the window for this bug
is pretty narrow (see sequence in [1] for details). Without adding sleeps it's
hard to cause a snapshot computation to happen between vacuum_set_xid_limits()
and lazy_scan_heap(). Cache invalidation processing needs to trigger
GetSnapshotData() below vac_open_indexes() (which in turn requires a catalog
invalidation to be received, followed by use of the catalog snapshot), and the
computed RecentXmin needs to change.
Except that it's not trivial to get right, I could see it being worthwhile to
add verification of hot chains to amcheck, and backpatch that to 14.
Greetings,
Andres Freund
[1] https://www.postgresql.org/message-id/20211118071941.e4far2w5vufnahun%40alap3.anarazel.de
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-02-04 15:15 Tom Lane <tgl@sss.pgh.pa.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 90+ messages in thread
From: Tom Lane @ 2022-02-04 15:15 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Peter Geoghegan <pg@bowt.ie>; Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Andres Freund <andres@anarazel.de> writes:
> On 2022-02-03 15:54:28 -0500, Tom Lane wrote:
>> I'm writing release notes and wondering what I can tell users about
>> how to detect or recover from this bug. Is a REINDEX sufficient,
>> or is the presence of the bogus redirect item going to cause
>> persistent problems?
> Good questions.
> It's hard to answer whether there's any danger after a REINDEX. Afaics the
> build scan would just pick the "lower offset" version of the root
> pointer. Which should be fine.
> It's possible there could be trouble down the line, e.g. heap pruning doing
> something weird once starting in a corrupted state, that then leads REINDEX to
> do something bogus. The simple cases look OK, because a second visit/action by
> heap_prune_chain for one tid from two different root pointers would see
> ->marked[offnum] as true. It gets more complicated once multiple intermediary
> row versions are involved, because the intermediary row versions won't be in
> ->marked if an entire chain is pruned. But afaict that should still end up
> looking like a hot chain ending in an aborted tuple or such.
OK, I'll just recommend REINDEX.
> Except that it's not trivial to get right, I could see it being worthwhile to
> add verification of hot chains to amcheck, and backpatch that to 14.
I'd have thought that'd be a fundamental component of a heap check
module, so +1 for adding it. Dunno about the back-patch part though.
It seems like a new feature.
regards, tom lane
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-04 00:22 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
2 siblings, 2 replies; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-04 00:22 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Dec 10, 2021 at 8:57 PM Andres Freund <andres@anarazel.de> wrote:
> 0004 is a patch that tries to address your point about the GlobalVisTestFor()
> placement, sort of
Following up on this now (will follow up on the "harden pruning" patch
separately).
> My earlier point that moving it to earlier would be a bad idea because it'd
> make the horizon "older" was bogus - GlobalVisTestFor() doesn't itself do
> any horizon determination. However moving the GlobalVisTestFor() earlier
> still seems wrong - imo the vacuum_set_xid_limits() call should be moved to
> later. The attached patch moves both, wrapped in a new function, to just
> before the scan in lazy_scan_heap().
Attached is (roughly speaking) a rebased version of this patch (i.e.
your v4-0004- patch from the patch series). Like your patch, this
patch documents our expectations for vistest and OldestXmin. Unlike
your patch, it makes all vacrel initialization take place before
lazy_scan_heap is called, from inside heap_vacuum_rel.
An important detail here is how OldestXmin (and vistest) are related
to rel_pages -- I also added something about that. It definitely
wouldn't be okay to (say) move the call to RelationGetNumberOfBlocks
to any place before the call to vacuum_set_xid_limits. And so it very
much seems in scope here.
I'm a bit confused about at least one detail here: you've said here
that "GlobalVisTestFor() doesn't itself do any horizon determination",
which seems to contradict comments from the v4-0004- patch that was
attached to the same email. This WIP patch claims that there is some
advantage to delaying the call to GlobalVisTestFor regardless, though
I think that there is a good chance that that detail is wrong. Please
let me know which it is.
I also updated nearby comments about the update to the target heap
rel's pg_class entry -- that work was done in passing.
Thanks
--
Peter Geoghegan
Attachments:
v1-0001-WIP-Move-GlobalVisState-state-into-vacrel.patch (application/octet-stream, 11.1K) download
From b6e569e51cc381bc19c5c28985bb180d7a6fd8c6 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Thu, 3 Mar 2022 13:28:34 -0800
Subject: [PATCH v1] WIP: Move GlobalVisState state into vacrel
---
src/backend/access/heap/vacuumlazy.c | 119 ++++++++++++++++-----------
1 file changed, 69 insertions(+), 50 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb..638ac5909 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -167,9 +167,10 @@ typedef struct LVRelState
MultiXactId relminmxid;
double old_live_tuples; /* previous value of pg_class.reltuples */
- /* VACUUM operation's cutoff for pruning */
+ /* VACUUM operation's cutoffs for freezing and pruning */
TransactionId OldestXmin;
- /* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
+ GlobalVisState *vistest;
+ /* VACUUM operation's target cutoffs for freezing XIDs and MultiXactIds */
TransactionId FreezeLimit;
MultiXactId MultiXactCutoff;
/* Are FreezeLimit/MultiXactCutoff still valid? */
@@ -185,8 +186,6 @@ typedef struct LVRelState
bool verbose; /* VACUUM VERBOSE? */
/*
- * State managed by lazy_scan_heap() follows.
- *
* dead_items stores TIDs whose index tuples are deleted by index
* vacuuming. Each TID points to an LP_DEAD line pointer from a heap page
* that has been processed by lazy_scan_prune. Also needed by
@@ -252,7 +251,6 @@ static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
bool sharelock, Buffer vmbuffer);
static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate);
static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
@@ -468,27 +466,65 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->relminmxid = rel->rd_rel->relminmxid;
vacrel->old_live_tuples = rel->rd_rel->reltuples;
- /* Set cutoffs for entire VACUUM */
+ /* Initialize counter fields explicitly (unnecessary, but be tidy) */
+ vacrel->scanned_pages = 0;
+ vacrel->frozenskipped_pages = 0;
+ vacrel->removed_pages = 0;
+ vacrel->lpdead_item_pages = 0;
+ vacrel->missed_dead_pages = 0;
+ vacrel->nonempty_pages = 0;
+ vacrel->num_index_scans = 0;
+ vacrel->tuples_deleted = 0;
+ vacrel->lpdead_items = 0;
+ vacrel->live_tuples = 0;
+ vacrel->recently_dead_tuples = 0;
+ vacrel->missed_dead_tuples = 0;
+
+ /* Allocate space for statistics output by index AMs */
+ vacrel->indstats = (IndexBulkDeleteResult **)
+ palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
+
+ /*
+ * Determine the extent of the blocks that we'll scan in lazy_scan_heap,
+ * and finalize cutoffs used for freezing and pruning in lazy_scan_prune.
+ *
+ * The OldestXmin and vistest cutoffs are almost equivalent: vistest will
+ * never fail to remove a deleted tuple with an xmax that is < OldestXmin.
+ * We need two separate cutoffs (one for freezing, another for pruning).
+ * We must ensure that heap_prune_page is always able to remove DEAD
+ * tuples (DEAD according to OldestXmin, not just RECENTLY_DEAD) so that
+ * lazy_scan_prune cannot ever become confused about whether it should
+ * freeze or remove any given tuple.
+ *
+ * Order matters here. We must call RelationGetNumberOfBlocks only after
+ * OldestXmin is established. Otherwise XIDs < OldestXmin could be missed
+ * from a page from after rel_pages. Also, we delay establishing vistest
+ * as a minor optimization. A later cutoff can enable more eager pruning.
+ */
+ orig_rel_pages = RelationGetNumberOfBlocks(rel);
vacrel->OldestXmin = OldestXmin;
+ vacrel->vistest = GlobalVisTestFor(rel);
+ /* FreezeLimit controls what tuples we freeze (always <= OldestXmin) */
vacrel->FreezeLimit = FreezeLimit;
vacrel->MultiXactCutoff = MultiXactCutoff;
+ vacrel->rel_pages = orig_rel_pages;
/* Track if cutoffs became invalid (possible in !aggressive case only) */
vacrel->freeze_cutoffs_valid = true;
/*
* Call lazy_scan_heap to perform all required heap pruning, index
- * vacuuming, and heap vacuuming (plus related processing)
+ * vacuuming, and heap vacuuming (plus related processing).
+ *
+ * Note: Resource management for parallel VACUUM is quite subtle, and so
+ * lazy_scan_heap must allocate (and deallocate) vacrel->dead_items space
+ * for itself. Every other vacrel field is initialized by now, though.
*/
lazy_scan_heap(vacrel, params->nworkers);
/* Done with indexes */
vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
- /*
- * Optionally truncate the relation. But remember the relation size used
- * by lazy_scan_heap for later first.
- */
- orig_rel_pages = vacrel->rel_pages;
+ /* Optionally truncate the relation */
if (should_attempt_truncation(vacrel))
{
update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
@@ -505,7 +541,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
PROGRESS_VACUUM_PHASE_FINAL_CLEANUP);
/*
- * Update statistics in pg_class.
+ * Prepare to update rel's pg_class entry.
+ *
+ * rel's indexes may have already been updated at end of lazy_scan_heap.
+ * That happens on a best-effort basis, since there is no need to maintain
+ * critical state like relfrozenxid. Indexes only need VACUUM to maintain
+ * reltuples and relpages statistics.
*
* In principle new_live_tuples could be -1 indicating that we (still)
* don't know the tuple count. In practice that probably can't happen,
@@ -523,6 +564,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
new_rel_allvisible = new_rel_pages;
/*
+ * Now actually update rel's pg_class entry.
+ *
* Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
* We are able to advance relfrozenxid in a non-aggressive VACUUM too,
* provided we didn't skip any all-visible (not all-frozen) pages using
@@ -787,7 +830,7 @@ static void
lazy_scan_heap(LVRelState *vacrel, int nworkers)
{
VacDeadItems *dead_items;
- BlockNumber nblocks,
+ BlockNumber rel_pages = vacrel->rel_pages,
blkno,
next_unskippable_block,
next_failsafe_block,
@@ -800,29 +843,6 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
PROGRESS_VACUUM_MAX_DEAD_TUPLES
};
int64 initprog_val[3];
- GlobalVisState *vistest;
-
- nblocks = RelationGetNumberOfBlocks(vacrel->rel);
- vacrel->rel_pages = nblocks;
- vacrel->scanned_pages = 0;
- vacrel->frozenskipped_pages = 0;
- vacrel->removed_pages = 0;
- vacrel->lpdead_item_pages = 0;
- vacrel->missed_dead_pages = 0;
- vacrel->nonempty_pages = 0;
-
- /* Initialize instrumentation counters */
- vacrel->num_index_scans = 0;
- vacrel->tuples_deleted = 0;
- vacrel->lpdead_items = 0;
- vacrel->live_tuples = 0;
- vacrel->recently_dead_tuples = 0;
- vacrel->missed_dead_tuples = 0;
-
- vistest = GlobalVisTestFor(vacrel->rel);
-
- vacrel->indstats = (IndexBulkDeleteResult **)
- palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
/*
* Do failsafe precheck before calling dead_items_alloc. This ensures
@@ -843,7 +863,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
- initprog_val[1] = nblocks;
+ initprog_val[1] = rel_pages;
initprog_val[2] = dead_items->max_items;
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
@@ -867,9 +887,9 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* Before entering the main loop, establish the invariant that
* next_unskippable_block is the next block number >= blkno that we can't
* skip based on the visibility map, either all-visible for a regular scan
- * or all-frozen for an aggressive scan. We set it to nblocks if there's
- * no such block. We also set up the skipping_blocks flag correctly at
- * this stage.
+ * or all-frozen for an aggressive scan. We set it to rel_pages when
+ * there's no such block. We also set up the skipping_blocks flag
+ * correctly at this stage.
*
* Note: The value returned by visibilitymap_get_status could be slightly
* out-of-date, since we make this test before reading the corresponding
@@ -887,7 +907,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block = 0;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmstatus;
@@ -914,7 +934,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
else
skipping_blocks = false;
- for (blkno = 0; blkno < nblocks; blkno++)
+ for (blkno = 0; blkno < rel_pages; blkno++)
{
Buffer buf;
Page page;
@@ -932,7 +952,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block++;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmskipflags;
@@ -977,7 +997,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/*
* The current page can be skipped if we've seen a long enough run
* of skippable blocks to justify skipping it -- provided it's not
- * the last page in the relation (according to rel_pages/nblocks).
+ * the last page in the relation (according to rel_pages).
*
* We always scan the table's last page to determine whether it
* has tuples or not, even if it would otherwise be skipped. This
@@ -985,7 +1005,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* on the table to attempt a truncation that just fails
* immediately because there are tuples on the last page.
*/
- if (skipping_blocks && blkno < nblocks - 1)
+ if (skipping_blocks && blkno < rel_pages - 1)
{
/*
* Tricky, tricky. If this is in aggressive vacuum, the page
@@ -1153,7 +1173,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* were pruned some time earlier. Also considers freezing XIDs in the
* tuple headers of remaining items with storage.
*/
- lazy_scan_prune(vacrel, buf, blkno, page, vistest, &prunestate);
+ lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
@@ -1352,7 +1372,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
vacrel->blkno = InvalidBlockNumber;
/* now we can compute the new value for pg_class.reltuples */
- vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
+ vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, rel_pages,
vacrel->scanned_pages,
vacrel->live_tuples);
@@ -1559,7 +1579,6 @@ lazy_scan_prune(LVRelState *vacrel,
Buffer buf,
BlockNumber blkno,
Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate)
{
Relation rel = vacrel->rel;
@@ -1598,7 +1617,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-04 03:31 Peter Geoghegan <pg@bowt.ie>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-04 03:31 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Mar 3, 2022 at 4:22 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Following up on this now (will follow up on the "harden pruning" patch
> separately).
Attached is a new revision of my fix. This is more or less a
combination of my v4 fix from November 12 [1] and Andres'
already-committed fix (commit 18b87b20), rebased on top of HEAD. This
patch was originally a bugfix, but now it's technically just
refactoring/hardening of the logic in pruneheap.c. It hasn't changed
all that much, though.
We're still doing an up-front scan of the heap page to precalculate
HTSV for each tuple (no change from commit 18b87b20), but we no longer
do that on correctness grounds. It is purely a performance thing now.
Note that I am making a working assumption that that still makes sense
for now (I think that it probably does, but I haven't yet verified it
for myself).
We now do "3 passes" over the page. The first is the aforementioned
"precalculate HTSV" pass, the second is for determining the extent of
HOT chains, and the third is for any remaining disconnected/orphaned
HOT chains. I suppose that that's okay, since the amount of work has
hardly increased in proportion to this "extra pass over the page". Not
100% sure about everything myself right now, though. I guess that "3
passes" might be considered excessive.
[1] https://postgr.es/m/CAH2-WzmNk6V6tqzuuabxoxM8HJRaWU6h12toaS-bqYcLiht16A@mail.gmail.com
--
Peter Geoghegan
Attachments:
v5-0001-Make-heap-pruning-more-robust.patch (application/octet-stream, 22.8K) download
From 32722e5837f6bbffb00f06a0ec5b92a6243a6e8a Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Thu, 3 Mar 2022 18:13:15 -0800
Subject: [PATCH v5] Make heap pruning more robust.
Follow-up to bugfix commit 18b87b20.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CAH2-WznNKY6ydUczuTXutVmb_dj3MnAcoaVYc8xyignWfNQ%3DFQ%40mail.gmail.com
Backpatch: master only (no backpatch)
---
src/backend/access/heap/pruneheap.c | 398 +++++++++++++++++-----------
1 file changed, 238 insertions(+), 160 deletions(-)
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index b6500763a..81b8fd62f 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -58,23 +58,24 @@ typedef struct
OffsetNumber nowdead[MaxHeapTuplesPerPage];
OffsetNumber nowunused[MaxHeapTuplesPerPage];
- /*
- * marked[i] is true if item i is entered in one of the above arrays.
- *
- * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
- * 1. Otherwise every access would need to subtract 1.
- */
- bool marked[MaxHeapTuplesPerPage + 1];
-
/*
* Tuple visibility is only computed once for each tuple, for correctness
* and efficiency reasons; see comment in heap_page_prune() for
* details. This is of type int8[,] intead of HTSV_Result[], so we can use
* -1 to indicate no visibility has been computed, e.g. for LP_DEAD items.
*
- * Same indexing as ->marked.
+ * This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
+ * 1. Otherwise every access would need to subtract 1.
*/
int8 htsv[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * visited[i] is true if item i was already visited by second pass over
+ * page (when we decide which tuples constitute each HOT chain).
+ *
+ * Same indexing as ->htsv.
+ */
+ bool visited[MaxHeapTuplesPerPage + 1];
} PruneState;
/* Local functions */
@@ -84,6 +85,8 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
PruneState *prstate);
+static int heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
@@ -293,32 +296,23 @@ heap_page_prune(Relation relation, Buffer buffer,
prstate.old_snap_xmin = old_snap_xmin;
prstate.old_snap_ts = old_snap_ts;
prstate.old_snap_used = false;
- prstate.latestRemovedXid = InvalidTransactionId;
- prstate.nredirected = prstate.ndead = prstate.nunused = 0;
- memset(prstate.marked, 0, sizeof(prstate.marked));
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
/*
- * Determine HTSV for all tuples.
+ * Determine HTSV for all tuples in first pass over page, and save it in
+ * prstate for later passes. Scan the page backwards (in reverse item
+ * offset number order).
*
- * This is required for correctness to deal with cases where running HTSV
- * twice could result in different results (e.g. RECENTLY_DEAD can turn to
- * DEAD if another checked item causes GlobalVisTestIsRemovableFullXid()
- * to update the horizon, INSERT_IN_PROGRESS can change to DEAD if the
- * inserting transaction aborts, ...). That in turn could cause
- * heap_prune_chain() to behave incorrectly if a tuple is reached twice,
- * once directly via a heap_prune_chain() and once following a HOT chain.
- *
- * It's also good for performance. Most commonly tuples within a page are
- * stored at decreasing offsets (while the items are stored at increasing
- * offsets). When processing all tuples on a page this leads to reading
- * memory at decreasing offsets within a page, with a variable stride.
- * That's hard for CPU prefetchers to deal with. Processing the items in
- * reverse order (and thus the tuples in increasing order) increases
- * prefetching efficiency significantly / decreases the number of cache
- * misses.
+ * This approach is good for performance. Most commonly tuples within a
+ * page are stored at decreasing offsets (while the items are stored at
+ * increasing offsets). When processing all tuples on a page this leads
+ * to reading memory at decreasing offsets within a page, with a variable
+ * stride. That's hard for CPU prefetchers to deal with. Processing the
+ * items in reverse order (and thus the tuples in increasing order)
+ * increases prefetching efficiency significantly / decreases the number
+ * of cache misses.
*/
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
@@ -350,30 +344,58 @@ heap_page_prune(Relation relation, Buffer buffer,
buffer);
}
- /* Scan the page */
+ /*
+ * Now scan the page a second time to process each HOT chain. This uses
+ * HTSV state saved by initial pass.
+ *
+ * It's possible that a few heap-only tuples will not get visited during
+ * this scan over the page. This happens when the tuples cannot be
+ * located by following a valid HOT chain.
+ */
+ prstate.latestRemovedXid = InvalidTransactionId;
+ prstate.nredirected = prstate.ndead = prstate.nunused = 0;
+ memset(prstate.visited, 0, sizeof(prstate.visited));
for (offnum = FirstOffsetNumber;
offnum <= maxoff;
offnum = OffsetNumberNext(offnum))
{
- ItemId itemid;
-
- /* Ignore items already processed as part of an earlier chain */
- if (prstate.marked[offnum])
+ /* Ignore items already visited as part of an earlier chain */
+ if (prstate.visited[offnum])
continue;
- /* see preceding loop */
+ /* see first scan/loop */
if (off_loc)
*off_loc = offnum;
- /* Nothing to do if slot is empty or already dead */
- itemid = PageGetItemId(page, offnum);
- if (!ItemIdIsUsed(itemid) || ItemIdIsDead(itemid))
- continue;
-
/* Process this item or chain of items */
ndeleted += heap_prune_chain(buffer, offnum, &prstate);
}
+ /*
+ * Now scan the page a third and final time. This is where we process
+ * disconnected heap-only tuples missed earlier on.
+ *
+ * These disconnected heap-only tuples (which always originate in aborted
+ * transactions) are always considered DEAD (and always become LP_UNUSED).
+ * Delaying processing of these tuples until here enables processing of
+ * HOT chains as whole units (that are known to be self-consistent).
+ */
+ for (offnum = FirstOffsetNumber;
+ offnum <= maxoff;
+ offnum = OffsetNumberNext(offnum))
+ {
+ /* Ignore items already visited (only want disconnected tuples) */
+ if (likely(prstate.visited[offnum]))
+ continue;
+
+ /* see first scan/loop */
+ if (off_loc)
+ *off_loc = offnum;
+
+ /* Process this disconnected heap-only tuple */
+ ndeleted += heap_prune_disconnected(buffer, offnum, &prstate);
+ }
+
/* Clear the offset information once we have processed the given page. */
if (off_loc)
*off_loc = InvalidOffsetNumber;
@@ -557,20 +579,22 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
/*
- * Prune specified line pointer or a HOT chain originating at line pointer.
+ * Prune HOT chain (or simple tuple) originating at specified line pointer.
*
- * If the item is an index-referenced tuple (i.e. not a heap-only tuple),
- * the HOT chain is pruned by removing all DEAD tuples at the start of the HOT
- * chain. We also prune any RECENTLY_DEAD tuples preceding a DEAD tuple.
- * This is OK because a RECENTLY_DEAD tuple preceding a DEAD tuple is really
- * DEAD, our visibility test is just too coarse to detect it.
+ * Used during second pass over the heap page (the root item pass). Won't
+ * process any heap-only tuples that cannot be found by traversing a HOT chain
+ * whose root item is at offset rootoffnum. Remaining "disconnected"
+ * heap-only tuples are dealt with in caller's third and final pass over the
+ * page instead. This is how we make sure that no DEAD tuples (or whole HOT
+ * chains) are missed during pruning.
*
* In general, pruning must never leave behind a DEAD tuple that still has
* tuple storage. VACUUM isn't prepared to deal with that case. That's why
* VACUUM prunes the same heap page a second time (without dropping its lock
* in the interim) when it sees a newly DEAD tuple that we initially saw as
- * in-progress. Retrying pruning like this can only happen when an inserting
- * transaction concurrently aborts.
+ * in-progress. Retrying pruning like this can only happen due to certain
+ * edge-cases, like the case where an inserting transaction concurrently
+ * aborts.
*
* The root line pointer is redirected to the tuple immediately after the
* latest DEAD tuple. If all tuples in the chain are DEAD, the root line
@@ -588,7 +612,6 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
{
- int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
TransactionId priorXmax = InvalidTransactionId;
ItemId rootlp;
@@ -596,63 +619,54 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
OffsetNumber latestdead = InvalidOffsetNumber,
maxoff = PageGetMaxOffsetNumber(dp),
offnum;
+ bool pastlatestdead = false;
OffsetNumber chainitems[MaxHeapTuplesPerPage];
- int nchain = 0,
- i;
+ int nchain;
+
+ Assert(!prstate->visited[rootoffnum]);
rootlp = PageGetItemId(dp, rootoffnum);
- /*
- * If it's a heap-only tuple, then it is not the start of a HOT chain.
- */
if (ItemIdIsNormal(rootlp))
{
Assert(prstate->htsv[rootoffnum] != -1);
htup = (HeapTupleHeader) PageGetItem(dp, rootlp);
+ /*
+ * If it's a heap-only tuple, then it is not the start of a HOT chain.
+ * We'll process it later, either by traversing its HOT chain
+ * (starting from the root item), or in heap_prune_disconnected() call
+ * made during our third pass over page.
+ */
if (HeapTupleHeaderIsHeapOnly(htup))
- {
- /*
- * If the tuple is DEAD and doesn't chain to anything else, mark
- * it unused immediately. (If it does chain, we can only remove
- * it as part of pruning its chain.)
- *
- * We need this primarily to handle aborted HOT updates, that is,
- * XMIN_INVALID heap-only tuples. Those might not be linked to by
- * any chain, since the parent tuple might be re-updated before
- * any pruning occurs. So we have to be able to reap them
- * separately from chain-pruning. (Note that
- * HeapTupleHeaderIsHotUpdated will never return true for an
- * XMIN_INVALID tuple, so this code will work even when there were
- * sequential updates within the aborted transaction.)
- *
- * Note that we might first arrive at a dead heap-only tuple
- * either here or while following a chain below. Whichever path
- * gets there first will mark the tuple unused.
- */
- if (prstate->htsv[rootoffnum] == HEAPTUPLE_DEAD &&
- !HeapTupleHeaderIsHotUpdated(htup))
- {
- heap_prune_record_unused(prstate, rootoffnum);
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- ndeleted++;
- }
+ return 0;
+ }
+ else if (!ItemIdIsRedirected(rootlp))
+ {
+ /*
+ * Nothing to do if slot cannot possibly be valid root item of HOT
+ * chain or a simple heap tuple
+ */
+ Assert(ItemIdIsDead(rootlp) || !ItemIdIsUsed(rootlp));
+ Assert(prstate->htsv[rootoffnum] == -1);
+ prstate->visited[rootoffnum] = true;
- /* Nothing more to do */
- return ndeleted;
- }
+ return 0;
}
- /* Start from the root tuple */
+ /*
+ * Start from the root item. Mark it as valid up front, since root items
+ * are always processed here (not as disconnected tuples in third pass
+ * over page).
+ */
+ prstate->visited[rootoffnum] = true;
offnum = rootoffnum;
+ nchain = 0;
/* while not end of the chain */
for (;;)
{
ItemId lp;
- bool tupdead,
- recent_dead;
/* Sanity check (pure paranoia) */
if (offnum < FirstOffsetNumber)
@@ -666,15 +680,11 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/* If item is already processed, stop --- it must not be same chain */
- if (prstate->marked[offnum])
+ if (nchain != 0 && prstate->visited[offnum])
break;
lp = PageGetItemId(dp, offnum);
- /* Unused item obviously isn't part of the chain */
- if (!ItemIdIsUsed(lp))
- break;
-
/*
* If we are looking at the redirected root line pointer, jump to the
* first normal tuple in the chain. If we find a redirect somewhere
@@ -689,13 +699,17 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
continue;
}
- /*
- * Likewise, a dead line pointer can't be part of the chain. (We
- * already eliminated the case of dead root tuple outside this
- * function.)
- */
- if (ItemIdIsDead(lp))
+ /* LP_UNUSED or LP_DEAD items obviously not part of the chain */
+ if (!ItemIdIsUsed(lp) || ItemIdIsDead(lp))
+ {
+ /*
+ * Can consider LP_UNUSED/LP_DEAD items visited when we arrive
+ * here by following a heap-only tuple's t_ctid link
+ */
+ Assert(prstate->htsv[rootoffnum] == -1);
+ prstate->visited[offnum] = true;
break;
+ }
Assert(ItemIdIsNormal(lp));
Assert(prstate->htsv[offnum] != -1);
@@ -709,33 +723,50 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
/*
- * OK, this tuple is indeed a member of the chain.
+ * OK, this tuple is indeed a member of the chain. Stick with this
+ * interpretation for the entire prune operation by marking the item
+ * as visited now.
*/
+ Assert((nchain == 0 && ItemIdIsNormal(rootlp)) ||
+ HeapTupleHeaderIsHeapOnly(htup));
chainitems[nchain++] = offnum;
+ prstate->visited[offnum] = true;
/*
* Check tuple's visibility status.
*/
- tupdead = recent_dead = false;
-
switch ((HTSV_Result) prstate->htsv[offnum])
{
case HEAPTUPLE_DEAD:
- tupdead = true;
+
+ /*
+ * Remember the offnum of the last DEAD tuple in this HOT
+ * chain. To keep things simple, don't treat heap-only tuples
+ * from a HOT chain as DEAD unless they're only preceded by
+ * other DEAD tuples (in addition to actually being DEAD).
+ * Remaining tuples that appear DEAD (but don't get treated as
+ * such by us) are from concurrently aborting updaters.
+ *
+ * VACUUM will ask us to prune the heap page again when it
+ * sees that there is a DEAD tuple left behind, but that would
+ * be necessary regardless of our approach here.
+ */
+ if (!pastlatestdead)
+ {
+ latestdead = offnum;
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ }
+ else
+ {
+ /* Deliberately don't call heap_prune_record_prunable() */
+ }
+
break;
case HEAPTUPLE_RECENTLY_DEAD:
- recent_dead = true;
-
- /*
- * This tuple may soon become DEAD. Update the hint field so
- * that the page is reconsidered for pruning in future.
- */
- heap_prune_record_prunable(prstate,
- HeapTupleHeaderGetUpdateXid(htup));
- break;
-
case HEAPTUPLE_DELETE_IN_PROGRESS:
+ pastlatestdead = true; /* no further DEAD tuples in CHAIN */
/*
* This tuple may soon become DEAD. Update the hint field so
@@ -747,6 +778,7 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
case HEAPTUPLE_LIVE:
case HEAPTUPLE_INSERT_IN_PROGRESS:
+ pastlatestdead = true; /* no further DEAD tuples in CHAIN */
/*
* If we wanted to optimize for aborts, we might consider
@@ -761,25 +793,12 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
break;
}
- /*
- * Remember the last DEAD tuple seen. We will advance past
- * RECENTLY_DEAD tuples just in case there's a DEAD one after them;
- * but we can't advance past anything else. We have to make sure that
- * we don't miss any DEAD tuples, since DEAD tuples that still have
- * tuple storage after pruning will confuse VACUUM.
- */
- if (tupdead)
- {
- latestdead = offnum;
- HeapTupleHeaderAdvanceLatestRemovedXid(htup,
- &prstate->latestRemovedXid);
- }
- else if (!recent_dead)
- break;
-
/*
* If the tuple is not HOT-updated, then we are at the end of this
- * HOT-update chain.
+ * HOT-update chain. There might actually be more tuples that were
+ * considered part of the same HOT chain in the past, before the
+ * updater's xact aborted. We'll process any such tuples later on
+ * instead, inside heap_prune_disconnected().
*/
if (!HeapTupleHeaderIsHotUpdated(htup))
break;
@@ -797,17 +816,29 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * If we found a DEAD tuple in the chain, adjust the HOT chain so that all
- * the DEAD tuples at the start of the chain are removed and the root line
- * pointer is appropriately redirected.
+ * Should never find an LP_REDIRECT root item that didn't already point to
+ * a valid item. While a heap-only tuple's t_ctid link can legitimately
+ * point to almost anything, the rules for LP_REDIRECT links are far
+ * stricter: LP_REDIRECTs must _always_ point to a valid heap-only tuple.
*/
+ Assert(ItemIdIsNormal(rootlp) ||
+ (ItemIdIsRedirected(rootlp) && nchain >= 2));
+
if (OffsetNumberIsValid(latestdead))
{
+ int i,
+ ndeleted = 0;
+
/*
- * Mark as unused each intermediate item that we are able to remove
- * from the chain.
+ * Okay, at least one tuple from the chain (or the single plain heap
+ * tuple) is considered DEAD. Record what to do with items in the
+ * chain now.
*
- * When the previous item is the last dead tuple seen, we are at the
+ * First deal with the non-root items from HOT chain. Mark earlier
+ * items we consider DEAD as LP_UNUSED (since they're heap-only
+ * tuples).
+ *
+ * When the previous item is the last DEAD tuple seen, we are at the
* right candidate for redirection.
*/
for (i = 1; (i < nchain) && (chainitems[i - 1] != latestdead); i++)
@@ -817,36 +848,91 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum, PruneState *prstate)
}
/*
- * If the root entry had been a normal tuple, we are deleting it, so
- * count it in the result. But changing a redirect (even to DEAD
- * state) doesn't count.
+ * If the root item is a normal tuple, we are logically deleting it,
+ * so count it in the result. But changing an LP_REDIRECT (even to
+ * make it LP_DEAD) doesn't get counted in ndeleted -- that would
+ * amount to double-counting DEAD tuples (with tuple storage) in
+ * ndeleted.
*/
if (ItemIdIsNormal(rootlp))
ndeleted++;
/*
+ * Finally, consider what to do with the root item itself.
+ *
* If the DEAD tuple is at the end of the chain, the entire chain is
- * dead and the root line pointer can be marked dead. Otherwise just
- * redirect the root to the correct chain member.
+ * considered DEAD. The root item must therefore become LP_DEAD.
+ * Otherwise just redirect the root to the correct chain member.
*/
if (i >= nchain)
heap_prune_record_dead(prstate, rootoffnum);
else
heap_prune_record_redirect(prstate, rootoffnum, chainitems[i]);
- }
- else if (nchain < 2 && ItemIdIsRedirected(rootlp))
- {
- /*
- * We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state.
- */
- heap_prune_record_dead(prstate, rootoffnum);
+
+ return ndeleted;
}
- return ndeleted;
+ return 0;
+}
+
+/*
+ * Handle disconnected heap-only tuples during third and final pass over page.
+ * We always expect to process these tuples as DEAD tuples here. Since
+ * they're heap-only tuples it follows that they'll always be set LP_UNUSED.
+ *
+ * This is how we handle aborted heap-only tuples that were not visited in our
+ * second pass (via HOT chain traversal with the usual cross-checks). These
+ * tuples occur when a parent tuple is updated, the updater aborts, and some
+ * unrelated updater re-updates the original parent tuple again. The parent's
+ * t_ctid link won't continue to point to the aborted tuple. (Even when it
+ * does, we won't consider the parent to have been HOT updated, just because
+ * its XMAX aborted -- so we still end up here for the aborted tuple).
+ *
+ * Like heap_prune_chain, we don't actually change the page here.
+ *
+ * Returns the number of tuples (to be) deleted from the page, though this
+ * should always be 1 in practice.
+*/
+static int
+heap_prune_disconnected(Buffer buffer, OffsetNumber offnum,
+ PruneState *prstate)
+{
+ Page dp = (Page) BufferGetPage(buffer);
+ ItemId lp;
+ HeapTupleHeader htup;
+
+ lp = PageGetItemId(dp, offnum);
+ Assert(ItemIdIsNormal(lp));
+ Assert(prstate->htsv[offnum] != -1);
+ htup = (HeapTupleHeader) PageGetItem(dp, lp);
+
+ /*
+ * Caller must make sure that the tuple at 'offnum' is in fact a heap-only
+ * tuple that is disconnected from its HOT chain
+ */
+ Assert(!prstate->visited[offnum]);
+ Assert(HeapTupleHeaderIsHeapOnly(htup));
+
+ /*
+ * We expect that disconnected heap-only tuples must be from aborted
+ * transactions. They must already be DEAD, or something is amiss.
+ */
+ if (likely((HTSV_Result) prstate->htsv[offnum] == HEAPTUPLE_DEAD))
+ {
+ heap_prune_record_unused(prstate, offnum);
+
+ /* Unnecessary, but be conservative here */
+ HeapTupleHeaderAdvanceLatestRemovedXid(htup,
+ &prstate->latestRemovedXid);
+ return 1;
+ }
+
+ /*
+ * Should always be DEAD. A DEAD heap-only tuple is always counted in
+ * top-level ndeleted counter for pruning operation.
+ */
+ Assert(false);
+ return 0;
}
/* Record lowest soon-prunable XID */
@@ -872,10 +958,6 @@ heap_prune_record_redirect(PruneState *prstate,
prstate->redirected[prstate->nredirected * 2] = offnum;
prstate->redirected[prstate->nredirected * 2 + 1] = rdoffnum;
prstate->nredirected++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
- Assert(!prstate->marked[rdoffnum]);
- prstate->marked[rdoffnum] = true;
}
/* Record line pointer to be marked dead */
@@ -885,8 +967,6 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
prstate->ndead++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
/* Record line pointer to be marked unused */
@@ -896,8 +976,6 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
Assert(prstate->nunused < MaxHeapTuplesPerPage);
prstate->nowunused[prstate->nunused] = offnum;
prstate->nunused++;
- Assert(!prstate->marked[offnum]);
- prstate->marked[offnum] = true;
}
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-10 00:25 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
1 sibling, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-03-10 00:25 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-03 16:22:11 -0800, Peter Geoghegan wrote:
> Attached is (roughly speaking) a rebased version of this patch (i.e.
> your v4-0004- patch from the patch series). Like your patch, this
> patch documents our expectations for vistest and OldestXmin. Unlike
> your patch, it makes all vacrel initialization take place before
> lazy_scan_heap is called, from inside heap_vacuum_rel.
I think it'd be nicer if we did the horizon determination after allocating
space for dead tuples... But it's still better than today, so...
> An important detail here is how OldestXmin (and vistest) are related
> to rel_pages -- I also added something about that. It definitely
> wouldn't be okay to (say) move the call to RelationGetNumberOfBlocks
> to any place before the call to vacuum_set_xid_limits. And so it very
> much seems in scope here.
You're right, it needs to be that way round.
> I'm a bit confused about at least one detail here: you've said here
> that "GlobalVisTestFor() doesn't itself do any horizon determination",
> which seems to contradict comments from the v4-0004- patch that was
> attached to the same email.
Which comment in the patch is the quoted statement contradicting? The on-point
comment seems to be
+ * [...] vacrel->vistest is always at least as aggressive as the
+ * limits vacuum_set_xid_limits() computes because ComputeXidHorizons()
+ * (via vacuum_set_xid_limits() ->GetOldestNonRemovableTransactionId())
+ * ensures the approximate horizons are always at least as aggressive as
+ * the precise horizons.
> + * Also, we delay establishing vistest + * as a minor optimization. A
> later cutoff can enable more eager pruning. + */
I don't think the minor optimization does anything (which I had stated wrongly
at some point in this thread).
> /*
> * Call lazy_scan_heap to perform all required heap pruning, index
> - * vacuuming, and heap vacuuming (plus related processing)
> + * vacuuming, and heap vacuuming (plus related processing).
> + *
> + * Note: Resource management for parallel VACUUM is quite subtle, and so
> + * lazy_scan_heap must allocate (and deallocate) vacrel->dead_items space
> + * for itself. Every other vacrel field is initialized by now, though.
> */
> lazy_scan_heap(vacrel, params->nworkers);
What are you referring to with "quite subtle"?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-10 00:46 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 0 replies; 90+ messages in thread
From: Andres Freund @ 2022-03-10 00:46 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-03 19:31:32 -0800, Peter Geoghegan wrote:
> Attached is a new revision of my fix. This is more or less a
> combination of my v4 fix from November 12 [1] and Andres'
> already-committed fix (commit 18b87b20), rebased on top of HEAD. This
> patch was originally a bugfix, but now it's technically just
> refactoring/hardening of the logic in pruneheap.c. It hasn't changed
> all that much, though.
Perhaps worth discussing outside of -bugs?
> We're still doing an up-front scan of the heap page to precalculate
> HTSV for each tuple (no change from commit 18b87b20), but we no longer
> do that on correctness grounds. It is purely a performance thing now.
> Note that I am making a working assumption that that still makes sense
> for now (I think that it probably does, but I haven't yet verified it
> for myself).
I'd be surprised if not.
> We now do "3 passes" over the page. The first is the aforementioned
> "precalculate HTSV" pass, the second is for determining the extent of
> HOT chains, and the third is for any remaining disconnected/orphaned
> HOT chains. I suppose that that's okay, since the amount of work has
> hardly increased in proportion to this "extra pass over the page". Not
> 100% sure about everything myself right now, though. I guess that "3
> passes" might be considered excessive.
We should be able to optimize away the third pass in the majority of cases, by
keeping track of the number of tuples visited or such. That seems like it
might be worth doing?
> + /*
> + * Start from the root item. Mark it as valid up front, since root items
> + * are always processed here (not as disconnected tuples in third pass
> + * over page).
> + */
> + prstate->visited[rootoffnum] = true;
> offnum = rootoffnum;
> + nchain = 0;
I wonder if it'd be clearer if we dealt with redirects outside of the
loop. Would make it easier to assert that the target of a redirect may not be
unused / !heap-only?
> +
> + /*
> + * Remember the offnum of the last DEAD tuple in this HOT
> + * chain. To keep things simple, don't treat heap-only tuples
> + * from a HOT chain as DEAD unless they're only preceded by
> + * other DEAD tuples (in addition to actually being DEAD).
s/actually/themselves/?
> + * Remaining tuples that appear DEAD (but don't get treated as
> + * such by us) are from concurrently aborting updaters.
I don't understand this bit. A DEAD tuple following a RECENTLY_DEAD one won't
be removed now, and doesn't need to involve a concurrent abort? Are you
thinking of "remaining" as the tuples not referenced in the previous sentences?
> + * VACUUM will ask us to prune the heap page again when it
> + * sees that there is a DEAD tuple left behind, but that would
> + * be necessary regardless of our approach here.
> + */
Only as long as we do another set of HTSV calls...
> case HEAPTUPLE_LIVE:
> case HEAPTUPLE_INSERT_IN_PROGRESS:
> + pastlatestdead = true; /* no further DEAD tuples in CHAIN */
If we don't do anything to the following tuples, why don't we just abort here?
I assume it is because we'd then treat them as disconnected? That should
probably be called out...
> - }
> - else if (nchain < 2 && ItemIdIsRedirected(rootlp))
> - {
> - /*
> - * We found a redirect item that doesn't point to a valid follow-on
> - * item. This can happen if the loop in heap_page_prune caused us to
> - * visit the dead successor of a redirect item before visiting the
> - * redirect item. We can clean up by setting the redirect item to
> - * DEAD state.
> - */
> - heap_prune_record_dead(prstate, rootoffnum);
> +
> + return ndeleted;
> }
Could there be such tuples from before a pg_upgrade? Do we need to deal with
them somehow?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 02:03 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-11 02:03 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Attached is v2 revision of the less critical vacuumlazy.c vistest
patch. I will respond to your second email about the pruneheap.c patch
on a new dedicated thread, per your suggestion.
NB: I intend to commit this revision of the patch (or something pretty
close to it) in the next few days, barring any objections.
On Wed, Mar 9, 2022 at 4:25 PM Andres Freund <andres@anarazel.de> wrote:
> I think it'd be nicer if we did the horizon determination after allocating
> space for dead tuples... But it's still better than today, so...
Why would it be nicer? Do you mean that it would be neater that way,
since every single field in vacrel gets initialized before the
lazy_scan_prune call, without exception?
I agree with that, but it doesn't seem important right now.
> > An important detail here is how OldestXmin (and vistest) are related
> > to rel_pages -- I also added something about that. It definitely
> > wouldn't be okay to (say) move the call to RelationGetNumberOfBlocks
> > to any place before the call to vacuum_set_xid_limits. And so it very
> > much seems in scope here.
>
> You're right, it needs to be that way round.
I think that this is well worth documenting. But it's also nice that
*almost* all initialization takes place inside heap_vacuum_rel() now,
before we call lazy_scan_heap().
> I don't think the minor optimization does anything (which I had stated wrongly
> at some point in this thread).
Got it. Comments that said that there was value in that are removed
from v2. But v2 does at least speculate about teaching lazy_scan_prune
to advance vistest as an optimization, from time to time -- which, as
you pointed out, is likely to be added in the future.
Talking about the optimization as future work makes sense, though not
because the future work is necessarily all that important. It just
seemed to me to be the easiest way of getting the following point
across to the reader: "vistest and OldestXmin are two distinct
representations of the same cutoff for now -- though don't make the
mistake of believing that they have to be the same cutoff".
> > /*
> > * Call lazy_scan_heap to perform all required heap pruning, index
> > - * vacuuming, and heap vacuuming (plus related processing)
> > + * vacuuming, and heap vacuuming (plus related processing).
> > + *
> > + * Note: Resource management for parallel VACUUM is quite subtle, and so
> > + * lazy_scan_heap must allocate (and deallocate) vacrel->dead_items space
> > + * for itself. Every other vacrel field is initialized by now, though.
> > */
> > lazy_scan_heap(vacrel, params->nworkers);
>
> What are you referring to with "quite subtle"?
There are 2 subtleties that I can think of offhand:
1. We deliberately do the initial lazy_check_wraparound_failsafe()
precheck before reaching dead_items_alloc(). Per the comments at the
start of lazy_scan_heap(), we won't use parallel VACUUM if the
failsafe kicks in during the precheck.
2. There is also deallocation to consider here: lazy_scan_heap () also
calls dead_items_cleanup(), which might have to release shared memory
-- which is actually handled by ending parallel mode.
dead_items_cleanup() must be called before update_index_statistics(),
which comes last of all -- since we can't update pg_class when in
parallel mode.
In v2 the same comment block now simply says that we have initialized
everything from vacrel except for vacrel->dead_items. It doesn't
explain why.
Thanks
--
Peter Geoghegan
Attachments:
v2-0001-vacuumlazy.c-Document-vistest-and-OldestXmin.patch (application/octet-stream, 11.5K) download
From 98c24db674571767b1272b430dae83d6babd39bf Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Thu, 3 Mar 2022 13:28:34 -0800
Subject: [PATCH v2] vacuumlazy.c: Document vistest and OldestXmin.
Explain the relationship between vacuumlazy.c's vistest cutoff (cutoff
used by pruning) and OldestXmin cutoff (cutoff used by freezing). Also
document a closely related assumption: we must establish the rel_pages
used by VACUUM's heap scan _after_ OldestXmin is established.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20211211045710.ljtuu4gfloh754rs@alap3.anarazel.de
Discussion: https://postgr.es/m/CAH2-WznYsUxVT156rCQ+q=YD4S4=1M37hWvvHLz-H1pwSM8-Ew@mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 121 ++++++++++++++++-----------
1 file changed, 71 insertions(+), 50 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb..37b42d50d 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -167,9 +167,10 @@ typedef struct LVRelState
MultiXactId relminmxid;
double old_live_tuples; /* previous value of pg_class.reltuples */
- /* VACUUM operation's cutoff for pruning */
+ /* VACUUM operation's cutoffs for freezing and pruning */
TransactionId OldestXmin;
- /* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
+ GlobalVisState *vistest;
+ /* VACUUM operation's target cutoffs for freezing XIDs and MultiXactIds */
TransactionId FreezeLimit;
MultiXactId MultiXactCutoff;
/* Are FreezeLimit/MultiXactCutoff still valid? */
@@ -185,8 +186,6 @@ typedef struct LVRelState
bool verbose; /* VACUUM VERBOSE? */
/*
- * State managed by lazy_scan_heap() follows.
- *
* dead_items stores TIDs whose index tuples are deleted by index
* vacuuming. Each TID points to an LP_DEAD line pointer from a heap page
* that has been processed by lazy_scan_prune. Also needed by
@@ -252,7 +251,6 @@ static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
bool sharelock, Buffer vmbuffer);
static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate);
static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
@@ -468,27 +466,67 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->relminmxid = rel->rd_rel->relminmxid;
vacrel->old_live_tuples = rel->rd_rel->reltuples;
- /* Set cutoffs for entire VACUUM */
+ /* Initialize page counters explicitly (be tidy) */
+ vacrel->scanned_pages = 0;
+ vacrel->frozenskipped_pages = 0;
+ vacrel->removed_pages = 0;
+ vacrel->lpdead_item_pages = 0;
+ vacrel->missed_dead_pages = 0;
+ vacrel->nonempty_pages = 0;
+
+ /* Allocate/initialize output statistics state */
+ vacrel->new_rel_tuples = 0;
+ vacrel->new_live_tuples = 0;
+ vacrel->indstats = (IndexBulkDeleteResult **)
+ palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
+
+ /* Initialize remaining counters (be tidy) */
+ vacrel->num_index_scans = 0;
+ vacrel->tuples_deleted = 0;
+ vacrel->lpdead_items = 0;
+ vacrel->live_tuples = 0;
+ vacrel->recently_dead_tuples = 0;
+ vacrel->missed_dead_tuples = 0;
+
+ /*
+ * Determine the extent of the blocks that we'll scan in lazy_scan_heap,
+ * and finalize cutoffs used for freezing and pruning in lazy_scan_prune.
+ *
+ * We expect vistest will always make heap_page_prune remove any deleted
+ * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
+ * confused about whether a tuple should be frozen or removed. (In the
+ * future we might want to teach lazy_scan_prune to recompute vistest from
+ * time to time, to increase the number of dead tuples it can prune away.)
+ *
+ * We must determine rel_pages _after_ OldestXmin has been established.
+ * Any XIDs < OldestXmin must at least be considered for freezing now
+ * (though perhaps not actually frozen). lazy_scan_heap's physical heap
+ * scan (scan of pages < rel_pages) is therefore guaranteed to not miss
+ * any XIDs of interest.
+ */
+ orig_rel_pages = RelationGetNumberOfBlocks(rel);
vacrel->OldestXmin = OldestXmin;
+ vacrel->vistest = GlobalVisTestFor(rel);
+ /* FreezeLimit indicates which tuples to freeze (always <= OldestXmin) */
vacrel->FreezeLimit = FreezeLimit;
vacrel->MultiXactCutoff = MultiXactCutoff;
+ vacrel->rel_pages = orig_rel_pages;
/* Track if cutoffs became invalid (possible in !aggressive case only) */
vacrel->freeze_cutoffs_valid = true;
/*
* Call lazy_scan_heap to perform all required heap pruning, index
- * vacuuming, and heap vacuuming (plus related processing)
+ * vacuuming, and heap vacuuming (plus related processing).
+ *
+ * Note: lazy_scan_heap allocates vacrel->dead_items space for itself.
+ * Everything else from vacrel has been initialized by now.
*/
lazy_scan_heap(vacrel, params->nworkers);
/* Done with indexes */
vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
- /*
- * Optionally truncate the relation. But remember the relation size used
- * by lazy_scan_heap for later first.
- */
- orig_rel_pages = vacrel->rel_pages;
+ /* Optionally truncate the relation */
if (should_attempt_truncation(vacrel))
{
update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
@@ -505,7 +543,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
PROGRESS_VACUUM_PHASE_FINAL_CLEANUP);
/*
- * Update statistics in pg_class.
+ * Prepare to update rel's pg_class entry.
+ *
+ * rel's indexes may have already been updated at end of lazy_scan_heap.
+ * That happens on a best-effort basis, since there is no need to maintain
+ * critical state like relfrozenxid. Indexes only need VACUUM to maintain
+ * reltuples and relpages statistics.
*
* In principle new_live_tuples could be -1 indicating that we (still)
* don't know the tuple count. In practice that probably can't happen,
@@ -523,6 +566,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
new_rel_allvisible = new_rel_pages;
/*
+ * Now actually update rel's pg_class entry.
+ *
* Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
* We are able to advance relfrozenxid in a non-aggressive VACUUM too,
* provided we didn't skip any all-visible (not all-frozen) pages using
@@ -787,7 +832,7 @@ static void
lazy_scan_heap(LVRelState *vacrel, int nworkers)
{
VacDeadItems *dead_items;
- BlockNumber nblocks,
+ BlockNumber rel_pages = vacrel->rel_pages,
blkno,
next_unskippable_block,
next_failsafe_block,
@@ -800,29 +845,6 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
PROGRESS_VACUUM_MAX_DEAD_TUPLES
};
int64 initprog_val[3];
- GlobalVisState *vistest;
-
- nblocks = RelationGetNumberOfBlocks(vacrel->rel);
- vacrel->rel_pages = nblocks;
- vacrel->scanned_pages = 0;
- vacrel->frozenskipped_pages = 0;
- vacrel->removed_pages = 0;
- vacrel->lpdead_item_pages = 0;
- vacrel->missed_dead_pages = 0;
- vacrel->nonempty_pages = 0;
-
- /* Initialize instrumentation counters */
- vacrel->num_index_scans = 0;
- vacrel->tuples_deleted = 0;
- vacrel->lpdead_items = 0;
- vacrel->live_tuples = 0;
- vacrel->recently_dead_tuples = 0;
- vacrel->missed_dead_tuples = 0;
-
- vistest = GlobalVisTestFor(vacrel->rel);
-
- vacrel->indstats = (IndexBulkDeleteResult **)
- palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
/*
* Do failsafe precheck before calling dead_items_alloc. This ensures
@@ -843,7 +865,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
- initprog_val[1] = nblocks;
+ initprog_val[1] = rel_pages;
initprog_val[2] = dead_items->max_items;
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
@@ -867,9 +889,9 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* Before entering the main loop, establish the invariant that
* next_unskippable_block is the next block number >= blkno that we can't
* skip based on the visibility map, either all-visible for a regular scan
- * or all-frozen for an aggressive scan. We set it to nblocks if there's
- * no such block. We also set up the skipping_blocks flag correctly at
- * this stage.
+ * or all-frozen for an aggressive scan. We set it to rel_pages when
+ * there's no such block. We also set up the skipping_blocks flag
+ * correctly at this stage.
*
* Note: The value returned by visibilitymap_get_status could be slightly
* out-of-date, since we make this test before reading the corresponding
@@ -887,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block = 0;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmstatus;
@@ -914,7 +936,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
else
skipping_blocks = false;
- for (blkno = 0; blkno < nblocks; blkno++)
+ for (blkno = 0; blkno < rel_pages; blkno++)
{
Buffer buf;
Page page;
@@ -932,7 +954,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block++;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmskipflags;
@@ -977,7 +999,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/*
* The current page can be skipped if we've seen a long enough run
* of skippable blocks to justify skipping it -- provided it's not
- * the last page in the relation (according to rel_pages/nblocks).
+ * the last page in the relation (according to rel_pages).
*
* We always scan the table's last page to determine whether it
* has tuples or not, even if it would otherwise be skipped. This
@@ -985,7 +1007,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* on the table to attempt a truncation that just fails
* immediately because there are tuples on the last page.
*/
- if (skipping_blocks && blkno < nblocks - 1)
+ if (skipping_blocks && blkno < rel_pages - 1)
{
/*
* Tricky, tricky. If this is in aggressive vacuum, the page
@@ -1153,7 +1175,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* were pruned some time earlier. Also considers freezing XIDs in the
* tuple headers of remaining items with storage.
*/
- lazy_scan_prune(vacrel, buf, blkno, page, vistest, &prunestate);
+ lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
@@ -1352,7 +1374,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
vacrel->blkno = InvalidBlockNumber;
/* now we can compute the new value for pg_class.reltuples */
- vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
+ vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, rel_pages,
vacrel->scanned_pages,
vacrel->live_tuples);
@@ -1559,7 +1581,6 @@ lazy_scan_prune(LVRelState *vacrel,
Buffer buf,
BlockNumber blkno,
Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate)
{
Relation rel = vacrel->rel;
@@ -1598,7 +1619,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 03:13 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 2 replies; 90+ messages in thread
From: Andres Freund @ 2022-03-11 03:13 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-10 18:03:46 -0800, Peter Geoghegan wrote:
> Attached is v2 revision of the less critical vacuumlazy.c vistest
> patch. I will respond to your second email about the pruneheap.c patch
> on a new dedicated thread, per your suggestion.
>
> NB: I intend to commit this revision of the patch (or something pretty
> close to it) in the next few days, barring any objections.
WFM.
> On Wed, Mar 9, 2022 at 4:25 PM Andres Freund <andres@anarazel.de> wrote:
> > I think it'd be nicer if we did the horizon determination after allocating
> > space for dead tuples... But it's still better than today, so...
>
> Why would it be nicer?
Large allocation can take a bit. Especially dead_item_alloc() sneakily
initializes parallelism (which is darn ugly). Determining the horizon after
doing expensive stuff gives you a slightly better horizon...
> /*
> + * Now actually update rel's pg_class entry.
> + *
> * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
> * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
> * provided we didn't skip any all-visible (not all-frozen) pages using
> @@ -787,7 +832,7 @@ static void
> lazy_scan_heap(LVRelState *vacrel, int nworkers)
> {
> VacDeadItems *dead_items;
> - BlockNumber nblocks,
> + BlockNumber rel_pages = vacrel->rel_pages,
> blkno,
> next_unskippable_block,
> next_failsafe_block,
> @@ -843,7 +865,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
>
> /* Report that we're scanning the heap, advertising total # of blocks */
> initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
> - initprog_val[1] = nblocks;
> + initprog_val[1] = rel_pages;
> initprog_val[2] = dead_items->max_items;
> pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
>
> @@ -867,9 +889,9 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> * Before entering the main loop, establish the invariant that
> * next_unskippable_block is the next block number >= blkno that we can't
> * skip based on the visibility map, either all-visible for a regular scan
> - * or all-frozen for an aggressive scan. We set it to nblocks if there's
> - * no such block. We also set up the skipping_blocks flag correctly at
> - * this stage.
> + * or all-frozen for an aggressive scan. We set it to rel_pages when
> + * there's no such block. We also set up the skipping_blocks flag
> + * correctly at this stage.
> *
> * Note: The value returned by visibilitymap_get_status could be slightly
> * out-of-date, since we make this test before reading the corresponding
> @@ -887,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> next_unskippable_block = 0;
> if (vacrel->skipwithvm)
> {
> - while (next_unskippable_block < nblocks)
> + while (next_unskippable_block < rel_pages)
> {
> uint8 vmstatus;
>
> @@ -914,7 +936,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> else
> skipping_blocks = false;
>
> - for (blkno = 0; blkno < nblocks; blkno++)
> + for (blkno = 0; blkno < rel_pages; blkno++)
> {
> Buffer buf;
> Page page;
> @@ -932,7 +954,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> next_unskippable_block++;
> if (vacrel->skipwithvm)
> {
> - while (next_unskippable_block < nblocks)
> + while (next_unskippable_block < rel_pages)
> {
> uint8 vmskipflags;
>
> @@ -977,7 +999,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> /*
> * The current page can be skipped if we've seen a long enough run
> * of skippable blocks to justify skipping it -- provided it's not
> - * the last page in the relation (according to rel_pages/nblocks).
> + * the last page in the relation (according to rel_pages).
> *
> * We always scan the table's last page to determine whether it
> * has tuples or not, even if it would otherwise be skipped. This
> @@ -985,7 +1007,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> * on the table to attempt a truncation that just fails
> * immediately because there are tuples on the last page.
> */
> - if (skipping_blocks && blkno < nblocks - 1)
> + if (skipping_blocks && blkno < rel_pages - 1)
> {
> /*
> * Tricky, tricky. If this is in aggressive vacuum, the page
> @@ -1153,7 +1175,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> * were pruned some time earlier. Also considers freezing XIDs in the
> * tuple headers of remaining items with storage.
> */
> - lazy_scan_prune(vacrel, buf, blkno, page, vistest, &prunestate);
> + lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
>
> Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
>
> @@ -1352,7 +1374,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
> vacrel->blkno = InvalidBlockNumber;
>
> /* now we can compute the new value for pg_class.reltuples */
> - vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
> + vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, rel_pages,
> vacrel->scanned_pages,
> vacrel->live_tuples);
The whole s/nblocks/rel_pages/ seems like it should be done separately.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 03:40 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-11 03:40 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Mar 10, 2022 at 7:13 PM Andres Freund <andres@anarazel.de> wrote:
> > NB: I intend to commit this revision of the patch (or something pretty
> > close to it) in the next few days, barring any objections.
>
> WFM.
Cool.
> > On Wed, Mar 9, 2022 at 4:25 PM Andres Freund <andres@anarazel.de> wrote:
> > > I think it'd be nicer if we did the horizon determination after allocating
> > > space for dead tuples... But it's still better than today, so...
> >
> > Why would it be nicer?
>
> Large allocation can take a bit. Especially dead_item_alloc() sneakily
> initializes parallelism (which is darn ugly). Determining the horizon after
> doing expensive stuff gives you a slightly better horizon...
I'm confused. You recently said "I don't think the minor optimization
[delaying establishing vistest] does anything (which I had stated
wrongly at some point in this thread)". But you now seem to be saying
that delaying establishing vistest has at least some small value as an
optimization. At least in theory.
> The whole s/nblocks/rel_pages/ seems like it should be done separately.
I'll break the mechanical s/nblocks/rel_pages/ out into a dedicated
commit, then.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 03:46 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-03-11 03:46 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-10 19:40:21 -0800, Peter Geoghegan wrote:
> On Thu, Mar 10, 2022 at 7:13 PM Andres Freund <andres@anarazel.de> wrote:
> > > On Wed, Mar 9, 2022 at 4:25 PM Andres Freund <andres@anarazel.de> wrote:
> > > > I think it'd be nicer if we did the horizon determination after allocating
> > > > space for dead tuples... But it's still better than today, so...
> > >
> > > Why would it be nicer?
> >
> > Large allocation can take a bit. Especially dead_item_alloc() sneakily
> > initializes parallelism (which is darn ugly). Determining the horizon after
> > doing expensive stuff gives you a slightly better horizon...
>
> I'm confused. You recently said "I don't think the minor optimization
> [delaying establishing vistest] does anything (which I had stated
> wrongly at some point in this thread)". But you now seem to be saying
> that delaying establishing vistest has at least some small value as an
> optimization. At least in theory.
I'm not talking about just moving the vistest acquisition, but also
vacuum_set_xid_limits(). Which obviously *does* benefit from delaying as long
as possible.
> > The whole s/nblocks/rel_pages/ seems like it should be done separately.
>
> I'll break the mechanical s/nblocks/rel_pages/ out into a dedicated
> commit, then.
Cool.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 03:52 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-11 03:52 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Mar 10, 2022 at 7:46 PM Andres Freund <andres@anarazel.de> wrote:
> I'm not talking about just moving the vistest acquisition, but also
> vacuum_set_xid_limits(). Which obviously *does* benefit from delaying as long
> as possible.
That sounds hard, or at least a lot of work given the benefits.
As the patch points out, we are required to establish rel_pages after
we have established OldestXmin. We *also* use rel_pages to determine
the size of the dead_items array -- we don't want to allocate space
that couldn't possibly be used (i.e. a dead_items array with room for
more than `MaxHeapTuplesPerPage * rel_pages` dead items in total).
In short, it will be necessary to break that dependency, somehow.
Which is possibly, certainly, but still quite messy.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 04:08 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-03-11 04:08 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-10 19:52:24 -0800, Peter Geoghegan wrote:
> As the patch points out, we are required to establish rel_pages after
> we have established OldestXmin.
Hm. Now that I think more about it, I'm not as convinced that's entirely true
anymore. We just need to make sure that we have an accurate "xid horizon" from
before determining the relation size, not an "xmin horizon" and use the Min()
of the two horizons. The "xid horizon" obviously is typically well ahead of
the "xmin horizon" (and always >=).
And even on the "xid horizon" I suspect something more aggressive could be
used.
But that's probably best done separately - but perhaps worth to "weaken" the
comment a bit?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 04:55 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-11 04:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Mar 10, 2022 at 8:08 PM Andres Freund <andres@anarazel.de> wrote:
> But that's probably best done separately - but perhaps worth to "weaken" the
> comment a bit?
I'm confused again.
I don't get why it's only going to be worth delaying establishing
vistest if we can also delay establishing OldestXmin in about the same
way. AFAICT the only compelling reason to have two separate cutoffs
from the point of view of vacuumlazy.c is that it enables periodically
refreshing vistest within lazy_scan_prune, to allow it to prune dead
tuples more eagerly (fewer recently dead tuples, more dead removed
tuples). That I can see, that much I get.
Periodically refreshing OldestXmin for freezing won't work, though. At
the very least it seems much less compelling than the vistest idea.
Yes, it probably is true that we could in principle further split
OldestXmin along "xmin vs xid horizon" lines (I should say "split it
again" -- you already split OldestXmin into "OldestXmin for freezing,
vistest for pruning" as part of your snapshot scalability work). But
would it really make any sense? If so, I can't see why. At least not
right now.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-11 18:42 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-11 18:42 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Thu, Mar 10, 2022 at 7:13 PM Andres Freund <andres@anarazel.de> wrote:
> Large allocation can take a bit. Especially dead_item_alloc() sneakily
> initializes parallelism (which is darn ugly).
Attached v3 revision of the patch makes this a bit clearer.
The patch now moves the pg_class updates for indexes from the end of
lazy_scan_heap() to its heap_vacuum_rel() caller, at the point just
after lazy_scan_heap() returns. This seems logical to me because it
makes all pg_class updates take place inside heap_vacuum_rel(). It
also means that we pretty start parallel mode right at the beginning
of lazy_scan_heap(), and end it right at the end -- which also seems
like a small improvement.
And so we no longer end parallel mode in order to be able to safely
update pg_class for indexes. Rather, we end parallel mode because
processing by lazy_scan_heap() as a whole completed.
--
Peter Geoghegan
Attachments: download all (2 files, .tar.gz)
v3-0002-vacuumlazy.c-Rename-nblocks-to-rel_pages.patch (application/octet-stream, 4.2K) download
From 3da780b85ff2bf4f8117d72711beb6c697ef7e40 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Thu, 10 Mar 2022 19:46:30 -0800
Subject: [PATCH v3 2/2] vacuumlazy.c: Rename nblocks to rel_pages.
It seems worth being consistent about referring to VACUUM's rel_pages
field consistently, so rename the local variable copy used by
lazy_scan_heap to rel_pages for consistency.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220311031351.sbge5m2bpvy2ttxg@alap3.anarazel.de
---
src/backend/access/heap/vacuumlazy.c | 22 +++++++++++-----------
1 file changed, 11 insertions(+), 11 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 79089622a..e72058873 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -829,7 +829,7 @@ static void
lazy_scan_heap(LVRelState *vacrel, int nworkers)
{
VacDeadItems *dead_items;
- BlockNumber nblocks = vacrel->rel_pages,
+ BlockNumber rel_pages = vacrel->rel_pages,
blkno,
next_unskippable_block,
next_failsafe_block,
@@ -862,7 +862,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
- initprog_val[1] = nblocks;
+ initprog_val[1] = rel_pages;
initprog_val[2] = dead_items->max_items;
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
@@ -886,9 +886,9 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* Before entering the main loop, establish the invariant that
* next_unskippable_block is the next block number >= blkno that we can't
* skip based on the visibility map, either all-visible for a regular scan
- * or all-frozen for an aggressive scan. We set it to nblocks if there's
- * no such block. We also set up the skipping_blocks flag correctly at
- * this stage.
+ * or all-frozen for an aggressive scan. We set it to rel_pages when
+ * there's no such block. We also set up the skipping_blocks flag
+ * correctly at this stage.
*
* Note: The value returned by visibilitymap_get_status could be slightly
* out-of-date, since we make this test before reading the corresponding
@@ -906,7 +906,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block = 0;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmstatus;
@@ -933,7 +933,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
else
skipping_blocks = false;
- for (blkno = 0; blkno < nblocks; blkno++)
+ for (blkno = 0; blkno < rel_pages; blkno++)
{
Buffer buf;
Page page;
@@ -951,7 +951,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_unskippable_block++;
if (vacrel->skipwithvm)
{
- while (next_unskippable_block < nblocks)
+ while (next_unskippable_block < rel_pages)
{
uint8 vmskipflags;
@@ -996,7 +996,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/*
* The current page can be skipped if we've seen a long enough run
* of skippable blocks to justify skipping it -- provided it's not
- * the last page in the relation (according to rel_pages/nblocks).
+ * the last page in the relation (according to rel_pages).
*
* We always scan the table's last page to determine whether it
* has tuples or not, even if it would otherwise be skipped. This
@@ -1004,7 +1004,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* on the table to attempt a truncation that just fails
* immediately because there are tuples on the last page.
*/
- if (skipping_blocks && blkno < nblocks - 1)
+ if (skipping_blocks && blkno < rel_pages - 1)
{
/*
* Tricky, tricky. If this is in aggressive vacuum, the page
@@ -1371,7 +1371,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
vacrel->blkno = InvalidBlockNumber;
/* now we can compute the new value for pg_class.reltuples */
- vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
+ vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, rel_pages,
vacrel->scanned_pages,
vacrel->live_tuples);
--
2.30.2
v3-0001-vacuumlazy.c-Document-vistest-and-OldestXmin.patch (application/octet-stream, 14.4K) download
From e7f71c4a643f0ece961d09daf811ef5bb32498b7 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Thu, 3 Mar 2022 13:28:34 -0800
Subject: [PATCH v3 1/2] vacuumlazy.c: Document vistest and OldestXmin.
Explain the relationship between vacuumlazy.c's vistest cutoff (cutoff
used by pruning) and OldestXmin cutoff (cutoff used by freezing). Also
document a closely related assumption: we must establish the rel_pages
used by VACUUM's heap scan _after_ OldestXmin is established.
Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20211211045710.ljtuu4gfloh754rs@alap3.anarazel.de
Discussion: https://postgr.es/m/CAH2-WznYsUxVT156rCQ+q=YD4S4=1M37hWvvHLz-H1pwSM8-Ew@mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 159 +++++++++++++++------------
1 file changed, 87 insertions(+), 72 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 40101e0cb..79089622a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -167,9 +167,10 @@ typedef struct LVRelState
MultiXactId relminmxid;
double old_live_tuples; /* previous value of pg_class.reltuples */
- /* VACUUM operation's cutoff for pruning */
+ /* VACUUM operation's cutoffs for freezing and pruning */
TransactionId OldestXmin;
- /* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
+ GlobalVisState *vistest;
+ /* VACUUM operation's target cutoffs for freezing XIDs and MultiXactIds */
TransactionId FreezeLimit;
MultiXactId MultiXactCutoff;
/* Are FreezeLimit/MultiXactCutoff still valid? */
@@ -185,8 +186,6 @@ typedef struct LVRelState
bool verbose; /* VACUUM VERBOSE? */
/*
- * State managed by lazy_scan_heap() follows.
- *
* dead_items stores TIDs whose index tuples are deleted by index
* vacuuming. Each TID points to an LP_DEAD line pointer from a heap page
* that has been processed by lazy_scan_prune. Also needed by
@@ -252,7 +251,6 @@ static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
bool sharelock, Buffer vmbuffer);
static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate);
static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
@@ -281,7 +279,7 @@ static void dead_items_alloc(LVRelState *vacrel, int nworkers);
static void dead_items_cleanup(LVRelState *vacrel);
static bool heap_page_is_all_visible(LVRelState *vacrel, Buffer buf,
TransactionId *visibility_cutoff_xid, bool *all_frozen);
-static void update_index_statistics(LVRelState *vacrel);
+static void update_relstats_all_indexes(LVRelState *vacrel);
static void vacuum_error_callback(void *arg);
static void update_vacuum_error_info(LVRelState *vacrel,
LVSavedErrInfo *saved_vacrel,
@@ -296,7 +294,8 @@ static void restore_vacuum_error_info(LVRelState *vacrel,
*
* This routine sets things up for and then calls lazy_scan_heap, where
* almost all work actually takes place. Finalizes everything after call
- * returns by managing rel truncation and updating pg_class statistics.
+ * returns by managing relation truncation and updating rel's pg_class
+ * entry. (Also updates pg_class entries for any indexes that need it.)
*
* At entry, we have already established a transaction and opened
* and locked the relation.
@@ -468,9 +467,51 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
vacrel->relminmxid = rel->rd_rel->relminmxid;
vacrel->old_live_tuples = rel->rd_rel->reltuples;
- /* Set cutoffs for entire VACUUM */
+ /* Initialize page counters explicitly (be tidy) */
+ vacrel->scanned_pages = 0;
+ vacrel->frozenskipped_pages = 0;
+ vacrel->removed_pages = 0;
+ vacrel->lpdead_item_pages = 0;
+ vacrel->missed_dead_pages = 0;
+ vacrel->nonempty_pages = 0;
+ /* dead_items_alloc allocates vacrel->dead_items later on */
+
+ /* Allocate/initialize output statistics state */
+ vacrel->new_rel_tuples = 0;
+ vacrel->new_live_tuples = 0;
+ vacrel->indstats = (IndexBulkDeleteResult **)
+ palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
+
+ /* Initialize remaining counters (be tidy) */
+ vacrel->num_index_scans = 0;
+ vacrel->tuples_deleted = 0;
+ vacrel->lpdead_items = 0;
+ vacrel->live_tuples = 0;
+ vacrel->recently_dead_tuples = 0;
+ vacrel->missed_dead_tuples = 0;
+
+ /*
+ * Determine the extent of the blocks that we'll scan in lazy_scan_heap,
+ * and finalize cutoffs used for freezing and pruning in lazy_scan_prune.
+ *
+ * We expect vistest will always make heap_page_prune remove any deleted
+ * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
+ * confused about whether a tuple should be frozen or removed. (In the
+ * future we might want to teach lazy_scan_prune to recompute vistest from
+ * time to time, to increase the number of dead tuples it can prune away.)
+ *
+ * We must determine rel_pages _after_ OldestXmin has been established.
+ * lazy_scan_heap's physical heap scan (scan of pages < rel_pages) is
+ * thereby guaranteed to not miss any tuples with XIDs < OldestXmin. These
+ * XIDs must at least be considered for freezing (though not necessarily
+ * frozen) during its scan.
+ */
+ vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
vacrel->OldestXmin = OldestXmin;
+ vacrel->vistest = GlobalVisTestFor(rel);
+ /* FreezeLimit controls XID freezing (always <= OldestXmin) */
vacrel->FreezeLimit = FreezeLimit;
+ /* MultiXactCutoff controls MXID freezing */
vacrel->MultiXactCutoff = MultiXactCutoff;
/* Track if cutoffs became invalid (possible in !aggressive case only) */
vacrel->freeze_cutoffs_valid = true;
@@ -481,21 +522,21 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
*/
lazy_scan_heap(vacrel, params->nworkers);
- /* Done with indexes */
+ /*
+ * Update pg_class entries for each of rel's indexes where appropriate.
+ *
+ * Unlike the later update to rel's pg_class entry, this is not critical.
+ * Maintains relpages/reltuples statistics used by the planner only.
+ */
+ if (vacrel->do_index_cleanup)
+ update_relstats_all_indexes(vacrel);
+
+ /* Done with rel's indexes */
vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
- /*
- * Optionally truncate the relation. But remember the relation size used
- * by lazy_scan_heap for later first.
- */
- orig_rel_pages = vacrel->rel_pages;
+ /* Optionally truncate rel */
if (should_attempt_truncation(vacrel))
- {
- update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
- vacrel->nonempty_pages,
- InvalidOffsetNumber);
lazy_truncate_heap(vacrel);
- }
/* Pop the error context stack */
error_context_stack = errcallback.previous;
@@ -505,7 +546,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
PROGRESS_VACUUM_PHASE_FINAL_CLEANUP);
/*
- * Update statistics in pg_class.
+ * Prepare to update rel's pg_class entry.
*
* In principle new_live_tuples could be -1 indicating that we (still)
* don't know the tuple count. In practice that probably can't happen,
@@ -517,12 +558,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
*/
new_rel_pages = vacrel->rel_pages; /* After possible rel truncation */
new_live_tuples = vacrel->new_live_tuples;
-
visibilitymap_count(rel, &new_rel_allvisible, NULL);
if (new_rel_allvisible > new_rel_pages)
new_rel_allvisible = new_rel_pages;
/*
+ * Now actually update rel's pg_class entry.
+ *
* Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
* We are able to advance relfrozenxid in a non-aggressive VACUUM too,
* provided we didn't skip any all-visible (not all-frozen) pages using
@@ -787,7 +829,7 @@ static void
lazy_scan_heap(LVRelState *vacrel, int nworkers)
{
VacDeadItems *dead_items;
- BlockNumber nblocks,
+ BlockNumber nblocks = vacrel->rel_pages,
blkno,
next_unskippable_block,
next_failsafe_block,
@@ -800,29 +842,6 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
PROGRESS_VACUUM_MAX_DEAD_TUPLES
};
int64 initprog_val[3];
- GlobalVisState *vistest;
-
- nblocks = RelationGetNumberOfBlocks(vacrel->rel);
- vacrel->rel_pages = nblocks;
- vacrel->scanned_pages = 0;
- vacrel->frozenskipped_pages = 0;
- vacrel->removed_pages = 0;
- vacrel->lpdead_item_pages = 0;
- vacrel->missed_dead_pages = 0;
- vacrel->nonempty_pages = 0;
-
- /* Initialize instrumentation counters */
- vacrel->num_index_scans = 0;
- vacrel->tuples_deleted = 0;
- vacrel->lpdead_items = 0;
- vacrel->live_tuples = 0;
- vacrel->recently_dead_tuples = 0;
- vacrel->missed_dead_tuples = 0;
-
- vistest = GlobalVisTestFor(vacrel->rel);
-
- vacrel->indstats = (IndexBulkDeleteResult **)
- palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
/*
* Do failsafe precheck before calling dead_items_alloc. This ensures
@@ -880,7 +899,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* might leave some dead tuples lying around, but the next vacuum will
* find them. But even when aggressive *is* set, it's still OK if we miss
* a page whose all-frozen marking has just been cleared. Any new XIDs
- * just added to that page are necessarily newer than the GlobalXmin we
+ * just added to that page are necessarily newer than the OldestXmin we
* computed, so they'll have no effect on the value to which we can safely
* set relfrozenxid. A similar argument applies for MXIDs and relminmxid.
*/
@@ -1153,7 +1172,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
* were pruned some time earlier. Also considers freezing XIDs in the
* tuple headers of remaining items with storage.
*/
- lazy_scan_prune(vacrel, buf, blkno, page, vistest, &prunestate);
+ lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
@@ -1392,15 +1411,11 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
lazy_cleanup_all_indexes(vacrel);
/*
- * Free resources managed by dead_items_alloc. This will end parallel
- * mode when needed (it must end before updating index statistics as we
- * can't write in parallel mode).
+ * Free resources managed by dead_items_alloc. This ends parallel mode in
+ * passing when necessary.
*/
dead_items_cleanup(vacrel);
-
- /* Update index statistics */
- if (vacrel->nindexes > 0 && vacrel->do_index_cleanup)
- update_index_statistics(vacrel);
+ Assert(!IsInParallelMode());
}
/*
@@ -1559,7 +1574,6 @@ lazy_scan_prune(LVRelState *vacrel,
Buffer buf,
BlockNumber blkno,
Page page,
- GlobalVisState *vistest,
LVPagePruneState *prunestate)
{
Relation rel = vacrel->rel;
@@ -1598,7 +1612,7 @@ retry:
* lpdead_items's final value can be thought of as the number of tuples
* that were deleted from indexes.
*/
- tuples_deleted = heap_page_prune(rel, buf, vistest,
+ tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
&vacrel->offnum);
@@ -2292,8 +2306,6 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
Assert(vacrel->nindexes > 0);
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
- Assert(TransactionIdIsNormal(vacrel->relfrozenxid));
- Assert(MultiXactIdIsValid(vacrel->relminmxid));
/* Precheck for XID wraparound emergencies */
if (lazy_check_wraparound_failsafe(vacrel))
@@ -2604,6 +2616,9 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
static bool
lazy_check_wraparound_failsafe(LVRelState *vacrel)
{
+ Assert(TransactionIdIsNormal(vacrel->relfrozenxid));
+ Assert(MultiXactIdIsValid(vacrel->relminmxid));
+
/* Don't warn more than once per VACUUM */
if (vacrel->failsafe_active)
return true;
@@ -2644,6 +2659,10 @@ lazy_check_wraparound_failsafe(LVRelState *vacrel)
static void
lazy_cleanup_all_indexes(LVRelState *vacrel)
{
+ double reltuples = vacrel->new_rel_tuples;
+ bool estimated_count = vacrel->scanned_pages < vacrel->rel_pages;
+
+ Assert(vacrel->do_index_cleanup);
Assert(vacrel->nindexes > 0);
/* Report that we are now cleaning up indexes */
@@ -2652,10 +2671,6 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
if (!ParallelVacuumIsActive(vacrel))
{
- double reltuples = vacrel->new_rel_tuples;
- bool estimated_count =
- vacrel->scanned_pages < vacrel->rel_pages;
-
for (int idx = 0; idx < vacrel->nindexes; idx++)
{
Relation indrel = vacrel->indrels[idx];
@@ -2669,9 +2684,9 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
else
{
/* Outsource everything to parallel variant */
- parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
+ parallel_vacuum_cleanup_all_indexes(vacrel->pvs, reltuples,
vacrel->num_index_scans,
- (vacrel->scanned_pages < vacrel->rel_pages));
+ estimated_count);
}
}
@@ -2797,10 +2812,6 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
* Also don't attempt it if we are doing early pruning/vacuuming, because a
* scan which cannot find a truncated heap page cannot determine that the
* snapshot is too old to read that page.
- *
- * This is split out so that we can test whether truncation is going to be
- * called for before we actually do it. If you change the logic here, be
- * careful to depend only on fields that lazy_scan_heap updates on-the-fly.
*/
static bool
should_attempt_truncation(LVRelState *vacrel)
@@ -2816,8 +2827,8 @@ should_attempt_truncation(LVRelState *vacrel)
possibly_freeable >= vacrel->rel_pages / REL_TRUNCATE_FRACTION) &&
old_snapshot_threshold < 0)
return true;
- else
- return false;
+
+ return false;
}
/*
@@ -2835,6 +2846,10 @@ lazy_truncate_heap(LVRelState *vacrel)
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_TRUNCATE);
+ /* Update error traceback information */
+ update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
+ vacrel->nonempty_pages, InvalidOffsetNumber);
+
/*
* Loop until no more truncating can be done.
*/
@@ -3328,13 +3343,13 @@ heap_page_is_all_visible(LVRelState *vacrel, Buffer buf,
* Update index statistics in pg_class if the statistics are accurate.
*/
static void
-update_index_statistics(LVRelState *vacrel)
+update_relstats_all_indexes(LVRelState *vacrel)
{
Relation *indrels = vacrel->indrels;
int nindexes = vacrel->nindexes;
IndexBulkDeleteResult **indstats = vacrel->indstats;
- Assert(!IsInParallelMode());
+ Assert(vacrel->do_index_cleanup);
for (int idx = 0; idx < nindexes; idx++)
{
--
2.30.2
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-12 21:49 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-03-12 21:49 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-10 20:55:37 -0800, Peter Geoghegan wrote:
> On Thu, Mar 10, 2022 at 8:08 PM Andres Freund <andres@anarazel.de> wrote:
> > But that's probably best done separately - but perhaps worth to "weaken" the
> > comment a bit?
>
> I'm confused again.
>
> I don't get why it's only going to be worth delaying establishing
> vistest if we can also delay establishing OldestXmin in about the same
> way.
GlobalVisTestFor() doesn't trawl through the procarray to compute horizons. It
just gets already computed horizons, potentially very approximate ones that
later will get updated when necessary to determine the visibility of a
concrete xid. You could call GlobalVisTestFor() and then later call call
vacuum_set_xid_limits() and the test by GlobalVisTestFor() would still be
accurate.
> AFAICT the only compelling reason to have two separate cutoffs from the
> point of view of vacuumlazy.c is that it enables periodically refreshing
> vistest within lazy_scan_prune, to allow it to prune dead tuples more
> eagerly (fewer recently dead tuples, more dead removed tuples). That I can
> see, that much I get.
The main reason to have vistest at all is that for on-access pruning computing
accurate horizons is excruciatingly expensive - the reason behind procarray
scaling badly were all the xmin accesses during normal snapshot computations!
And due to the way the pruning code is used for both vacuuming and on-access
pruning that kind of bleeds into vacuumlazy.c. That could have been reduced,
but it seemed beneficial to allow to update horizons as we go.
> Periodically refreshing OldestXmin for freezing won't work, though. At
> the very least it seems much less compelling than the vistest idea.
I think there's a fair bit of value in using an "as aggressive as possible"
initial OldestXmin. Which is used to set a bunch of cutoffs / determine
aggressiveness etc.
Once we compute "measured" relfrozenxid however, there's afaict otherwise not
much point in updating OldestXmin as we go, at least if we start to use
vistest for the horizon determinations inside vacuumlazy.c.
> Yes, it probably is true that we could in principle further split
> OldestXmin along "xmin vs xid horizon" lines (I should say "split it
> again" -- you already split OldestXmin into "OldestXmin for freezing,
> vistest for pruning" as part of your snapshot scalability work). But
> would it really make any sense? If so, I can't see why. At least not
> right now.
If we don't switch to using vistest for HTSV determinations in vacuumlazy.c,
then I don't think we can refresh vistest without causing problems in
lazy_scan_prune().
But I think refreshing horizons is a different discussion from using
as-recent-as-possible initial horizons...
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-14 19:05 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-14 19:05 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Sat, Mar 12, 2022 at 1:49 PM Andres Freund <andres@anarazel.de> wrote:
> GlobalVisTestFor() doesn't trawl through the procarray to compute horizons. It
> just gets already computed horizons, potentially very approximate ones that
> later will get updated when necessary to determine the visibility of a
> concrete xid.
That much is easy to understand. It's the most significant way in
which the snapshot scalability work improves scalability.
> You could call GlobalVisTestFor() and then later call call
> vacuum_set_xid_limits() and the test by GlobalVisTestFor() would still be
> accurate.
That seems kinda obvious, since all it does is get the appropriate
state for the relkind of the relation being vacuumed. Typically it
just returns a pointer to GlobalVisDataRels, which is a static
variable in procarray.c.
> The main reason to have vistest at all is that for on-access pruning computing
> accurate horizons is excruciatingly expensive - the reason behind procarray
> scaling badly were all the xmin accesses during normal snapshot computations!
> And due to the way the pruning code is used for both vacuuming and on-access
> pruning that kind of bleeds into vacuumlazy.c. That could have been reduced,
> but it seemed beneficial to allow to update horizons as we go.
That I get too.
> > Periodically refreshing OldestXmin for freezing won't work, though. At
> > the very least it seems much less compelling than the vistest idea.
>
> I think there's a fair bit of value in using an "as aggressive as possible"
> initial OldestXmin. Which is used to set a bunch of cutoffs / determine
> aggressiveness etc.
I just don't think that it makes very much difference, since we're
only talking about freezing here -- which is not something we tend to
be too eager with anyway (FreezeLimit is very seldom the same as
OldestXmin, etc).
Maybe it could take a long time for vac_open_indexes() to return, but
that's really an edge case. I'm puzzled why you're placing so much
emphasis on this. I can see why that's valuable in a theoretical
abstract kind of way -- but that's about it. And so it feels like I'm
still missing something.
> Once we compute "measured" relfrozenxid however, there's afaict otherwise not
> much point in updating OldestXmin as we go, at least if we start to use
> vistest for the horizon determinations inside vacuumlazy.c.
> If we don't switch to using vistest for HTSV determinations in vacuumlazy.c,
> then I don't think we can refresh vistest without causing problems in
> lazy_scan_prune().
Why? Right now we could easily have a concurrent opportunistic prune
that uses a vistest that's well ahead of the one in VACUUM. And so
AFAICT it's already quite possible that any page encountered within
lazy_scan_prune was already pruned using a much more recent
vistest. So what's the problem with doing the same thing inside the
backend running VACUUM instead?
Is it something to do with the VACUUM backend's state?
We can recompute backend horizons during VACUUM after OldestXmin is
established, so I can't see why the backend's procarray.c state should
matter. If it did that would seem like a big problem in general. In
particular, my commit 9dd963ae25 (an nbtree VACUUM enhancement)
independently calls GetOldestNonRemovableTransactionId() midway
through VACUUM.
> But I think refreshing horizons is a different discussion from using
> as-recent-as-possible initial horizons...
Agreed. This is why I find your emphasis on as-recent-as-possible
initial horizons confusing. It seems as if you're giving almost equal
emphasis to both issues, even though one issue (more eager pruning
during VACUUM) is of obvious practical value, while the other
(as-recent-as-possible initial horizons) is pretty theoretical and
academic -- at best.
--
Peter Geoghegan
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-18 14:53 Andres Freund <andres@anarazel.de>
parent: Peter Geoghegan <pg@bowt.ie>
0 siblings, 1 reply; 90+ messages in thread
From: Andres Freund @ 2022-03-18 14:53 UTC (permalink / raw)
To: Peter Geoghegan <pg@bowt.ie>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Hi,
On 2022-03-14 12:05:14 -0700, Peter Geoghegan wrote:
> On Sat, Mar 12, 2022 at 1:49 PM Andres Freund <andres@anarazel.de> wrote:
> > > Periodically refreshing OldestXmin for freezing won't work, though. At
> > > the very least it seems much less compelling than the vistest idea.
> >
> > I think there's a fair bit of value in using an "as aggressive as possible"
> > initial OldestXmin. Which is used to set a bunch of cutoffs / determine
> > aggressiveness etc.
>
> I just don't think that it makes very much difference, since we're
> only talking about freezing here -- which is not something we tend to
> be too eager with anyway (FreezeLimit is very seldom the same as
> OldestXmin, etc).
Well, it's not uncommon to VACUUM FREEZE after ETL etc. But more importantly,
the concrete OldestXmin value matters, because it's the difference between
removing or keeping dead rows.
> Maybe it could take a long time for vac_open_indexes() to return, but
> that's really an edge case. I'm puzzled why you're placing so much
> emphasis on this. I can see why that's valuable in a theoretical
> abstract kind of way -- but that's about it. And so it feels like I'm
> still missing something.
I'm not saying it's a *huge* improvement. But especially for smaller tables
it's not uncommon to see multiple autovacuums on a table in quick succession
just because of a slightly outdated horizon. Of course you're not going to see
that in a workload with a handfull of heavily modified tables, but if there's
many tables it's a different story.
> > Once we compute "measured" relfrozenxid however, there's afaict otherwise not
> > much point in updating OldestXmin as we go, at least if we start to use
> > vistest for the horizon determinations inside vacuumlazy.c.
>
> > If we don't switch to using vistest for HTSV determinations in vacuumlazy.c,
> > then I don't think we can refresh vistest without causing problems in
> > lazy_scan_prune().
>
> Why? Right now we could easily have a concurrent opportunistic prune
> that uses a vistest that's well ahead of the one in VACUUM. And so
> AFAICT it's already quite possible that any page encountered within
> lazy_scan_prune was already pruned using a much more recent
> vistest. So what's the problem with doing the same thing inside the
> backend running VACUUM instead?
That was a brainfart on my end...
> > But I think refreshing horizons is a different discussion from using
> > as-recent-as-possible initial horizons...
>
> Agreed. This is why I find your emphasis on as-recent-as-possible
> initial horizons confusing. It seems as if you're giving almost equal
> emphasis to both issues, even though one issue (more eager pruning
> during VACUUM) is of obvious practical value, while the other
> (as-recent-as-possible initial horizons) is pretty theoretical and
> academic -- at best.
I'm not intending to weigh them the same. I think using a more recent horizon
is more important than you describe here, but "computed horizons" will be a
considerably larger win.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 90+ messages in thread
* Re: BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum
@ 2022-03-18 17:06 Peter Geoghegan <pg@bowt.ie>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 90+ messages in thread
From: Peter Geoghegan @ 2022-03-18 17:06 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Dmitry Dolgov <9erthalion6@gmail.com>; Alexander Lakhin <exclusion@gmail.com>; Matthias van de Meent <boekewurm+postgres@gmail.com>; PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Mar 18, 2022 at 7:53 AM Andres Freund <andres@anarazel.de> wrote:
> > Agreed. This is why I find your emphasis on as-recent-as-possible
> > initial horizons confusing. It seems as if you're giving almost equal
> > emphasis to both issues, even though one issue (more eager pruning
> > during VACUUM) is of obvious practical value, while the other
> > (as-recent-as-possible initial horizons) is pretty theoretical and
> > academic -- at best.
>
> I'm not intending to weigh them the same. I think using a more recent horizon
> is more important than you describe here, but "computed horizons" will be a
> considerably larger win.
Okay, got it.
My relfrozenxid and freezing patch series patch v10-0002-* took a new
approach to avoiding the visibility map aggressive VACUUM special
case, that seems like it might be relevant here (surprisingly enough).
The latest version totally embraces making all visibility map access
inside lazy_scan_heap use local state describing a range of skippable
blocks. Not that different, really, but still something that hints at
an interesting future direction.
What if we built a palloc'd array of these ranges up front, right
after establishing OldestXmin? We could even build a complete picture
of the entire table before we'd even scanned the first block. I don't
think it would require too much space. We'd then be operating against
a "snapshot of the visibility map just after OldestXmin was
established", even in a multi-hour VACUUM operation. This is kind of a
move in the opposite direction, but it's not necessarily the wrong
direction; at least this way we're not going to dirty concurrently
modified pages from the largest tables, just to set hint bits or clean
up one or two concurrently aborted transactions.
I also think that this would help with your aio work. Having a
palloc'd array like this would be an ideal target for prefetching. I
actually checked what you were doing here. Evidently the visibility
map stuff at the top of lazy_scan_heap was already a problem for aio:
https://github.com/anarazel/postgres/blob/29d3a3a3ddd47906d9128bc68c218664ab15a2c6/src/backend/acces...
I'm posting a more refined version of just that patch here, since it's
closer to the ideal for this kind of thing; it takes all of the
decisions away from lazy_scan_heap, which need only execute skipping
by following instructions built by the new helper function.
--
Peter Geoghegan
Attachments:
v1010-0002-Generalize-how-VACUUM-skips-all-frozen-pages.patch (application/x-patch, 19.6K) download
From 078f402ae5e2c73cc2e856eeea128c31fafafb75 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Fri, 11 Mar 2022 19:16:02 -0800
Subject: [PATCH v1010 2/9] Generalize how VACUUM skips all-frozen pages.
Non-aggressive VACUUMs were at a gratuitous disadvantage (relative to
aggressive VACUUMs) around advancing relfrozenxid before now. The
underlying issue was that lazy_scan_heap conditioned its skipping
behavior on whether or not the current VACUUM was aggressive. VACUUM
could fail to increment its frozenskipped_pages counter as a result, and
so could miss out on advancing relfrozenxid (in the non-aggressive case)
for no good reason.
The issue only comes up when concurrent activity might unset a page's
visibility map bit at exactly the wrong time. The non-aggressive case
rechecked the visibility map at the point of skipping each page before
now. This created a window for some other session to concurrently unset
the same heap page's bit in the visibility map. If the bit was unset at
the wrong time, it would cause VACUUM to conservatively conclude that
the page was _never_ all-frozen on recheck. frozenskipped_pages would
not be incremented for the page as a result. lazy_scan_heap had already
committed to skipping the page/range at that point, though -- which made
it unsafe to advance relfrozenxid/relminmxid later on.
Consistently avoid the issue by generalizing how we skip frozen pages
during aggressive VACUUMs: take the same approach when skipping any
skippable page range during aggressive and non-aggressive VACUUMs alike.
The new approach makes ranges (not individual pages) the fundamental
unit of skipping using the visibility map. frozenskipped_pages is
replaced with a boolean flag that represents whether some skippable
range with one or more all-visible pages was actually skipped (making
relfrozenxid unsafe to update). The VM_ALL_VISIBLE()/VM_ALL_FROZEN()
rechecks at the top of lazy_scan_heap are no longer required, since we
now record the same information in the book keeping state that tracks
the range as a whole.
There is now a clean and unambiguous separation between deciding which
contiguous pages are safe to skip (and so should be treated as a range
of skippable pages), deciding if it's worth skipping a given range, and
actually executing skipping. This separation seems like it might be
useful in the future. For example, it would now be straightforward to
teach VACUUM to assemble skippable ranges up front, via a batch process.
Many unprocessed skippable ranges could be stored in a palloc'd array;
there are no dependencies to complicate things for lazy_scan_heap later.
It's possible that the issue this commit fixes hardly ever came up in
practice. But we only had to be unlucky once to lose out on advancing
relfrozenxid -- a single affected heap page was enough to throw VACUUM
off. That seems like something to avoid on general principle. This is
similar to an issue fixed by commit 44fa8488, which taught vacuumlazy.c
to not give up on non-aggressive relfrozenxid advancement just because a
cleanup lock wasn't immediately available on some heap page.
Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-Wzn6bGJGfOy3zSTJicKLw99PHJeSOQBOViKjSCinaxUKDQ@mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 313 +++++++++++++--------------
1 file changed, 148 insertions(+), 165 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index ae280d4f9..cee4bf14a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -176,6 +176,7 @@ typedef struct LVRelState
/* Tracks oldest extant XID/MXID for setting relfrozenxid/relminmxid */
TransactionId NewRelfrozenXid;
MultiXactId NewRelminMxid;
+ bool skippedallvis;
/* Error reporting state */
char *relnamespace;
@@ -196,7 +197,6 @@ typedef struct LVRelState
VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
- BlockNumber frozenskipped_pages; /* # frozen pages skipped via VM */
BlockNumber removed_pages; /* # pages removed by relation truncation */
BlockNumber lpdead_item_pages; /* # pages with LP_DEAD items */
BlockNumber missed_dead_pages; /* # pages with missed dead tuples */
@@ -247,6 +247,10 @@ typedef struct LVSavedErrInfo
/* non-export function prototypes */
static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
+static BlockNumber lazy_scan_skip_range(LVRelState *vacrel, Buffer *vmbuffer,
+ BlockNumber next_block,
+ bool *all_visible_next_unskippable,
+ bool *skipping_range);
static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
BlockNumber blkno, Page page,
bool sharelock, Buffer vmbuffer);
@@ -471,7 +475,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
/* Initialize page counters explicitly (be tidy) */
vacrel->scanned_pages = 0;
- vacrel->frozenskipped_pages = 0;
vacrel->removed_pages = 0;
vacrel->lpdead_item_pages = 0;
vacrel->missed_dead_pages = 0;
@@ -518,6 +521,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
/* Initialize state used to track oldest extant XID/XMID */
vacrel->NewRelfrozenXid = OldestXmin;
vacrel->NewRelminMxid = OldestMxact;
+ vacrel->skippedallvis = false;
/*
* Call lazy_scan_heap to perform all required heap pruning, index
@@ -575,7 +579,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* to an XID that is either older or newer than FreezeLimit (same applies
* to relminmxid and MultiXactCutoff).
*/
- if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
+ if (vacrel->skippedallvis)
{
/* Skipped an all-visible page, so cannot advance relfrozenxid */
Assert(!aggressive);
@@ -587,8 +591,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
else
{
- Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
- orig_rel_pages);
Assert(!aggressive ||
TransactionIdPrecedesOrEquals(FreezeLimit,
vacrel->NewRelfrozenXid));
@@ -841,7 +843,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
next_failsafe_block,
next_fsm_block_to_vacuum;
Buffer vmbuffer = InvalidBuffer;
- bool skipping_blocks;
+ bool skipping_range,
+ all_visible_next_unskippable;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
@@ -872,179 +875,52 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
initprog_val[2] = dead_items->max_items;
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
- /*
- * Set things up for skipping blocks using visibility map.
- *
- * Except when vacrel->aggressive is set, we want to skip pages that are
- * all-visible according to the visibility map, but only when we can skip
- * at least SKIP_PAGES_THRESHOLD consecutive pages. Since we're reading
- * sequentially, the OS should be doing readahead for us, so there's no
- * gain in skipping a page now and then; that's likely to disable
- * readahead and so be counterproductive. Also, skipping even a single
- * page means that we can't update relfrozenxid, so we only want to do it
- * if we can skip a goodly number of pages.
- *
- * When vacrel->aggressive is set, we can't skip pages just because they
- * are all-visible, but we can still skip pages that are all-frozen, since
- * such pages do not need freezing and do not affect the value that we can
- * safely set for relfrozenxid or relminmxid.
- *
- * Before entering the main loop, establish the invariant that
- * next_unskippable_block is the next block number >= blkno that we can't
- * skip based on the visibility map, either all-visible for a regular scan
- * or all-frozen for an aggressive scan. We set it to rel_pages when
- * there's no such block. We also set up the skipping_blocks flag
- * correctly at this stage.
- *
- * Note: The value returned by visibilitymap_get_status could be slightly
- * out-of-date, since we make this test before reading the corresponding
- * heap page or locking the buffer. This is OK. If we mistakenly think
- * that the page is all-visible or all-frozen when in fact the flag's just
- * been cleared, we might fail to vacuum the page. It's easy to see that
- * skipping a page when aggressive is not set is not a very big deal; we
- * might leave some dead tuples lying around, but the next vacuum will
- * find them. But even when aggressive *is* set, it's still OK if we miss
- * a page whose all-frozen marking has just been cleared. Any new XIDs
- * just added to that page are necessarily >= vacrel->OldestXmin, and so
- * they cannot invalidate NewRelfrozenXid tracking. A similar argument
- * applies for NewRelminMxid tracking and OldestMxact.
- */
- next_unskippable_block = 0;
- if (vacrel->skipwithvm)
- {
- while (next_unskippable_block < rel_pages)
- {
- uint8 vmstatus;
-
- vmstatus = visibilitymap_get_status(vacrel->rel,
- next_unskippable_block,
- &vmbuffer);
- if (vacrel->aggressive)
- {
- if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
- break;
- }
- else
- {
- if ((vmstatus & VISIBILITYMAP_ALL_VISIBLE) == 0)
- break;
- }
- vacuum_delay_point();
- next_unskippable_block++;
- }
- }
-
- if (next_unskippable_block >= SKIP_PAGES_THRESHOLD)
- skipping_blocks = true;
- else
- skipping_blocks = false;
-
+ /* Set up an initial range of blocks to skip via the visibility map */
+ next_unskippable_block = lazy_scan_skip_range(vacrel, &vmbuffer, 0,
+ &all_visible_next_unskippable,
+ &skipping_range);
for (blkno = 0; blkno < rel_pages; blkno++)
{
Buffer buf;
Page page;
- bool all_visible_according_to_vm = false;
+ bool all_visible_according_to_vm;
LVPagePruneState prunestate;
- pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
-
- update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
- blkno, InvalidOffsetNumber);
-
if (blkno == next_unskippable_block)
{
- /* Time to advance next_unskippable_block */
- next_unskippable_block++;
- if (vacrel->skipwithvm)
- {
- while (next_unskippable_block < rel_pages)
- {
- uint8 vmskipflags;
-
- vmskipflags = visibilitymap_get_status(vacrel->rel,
- next_unskippable_block,
- &vmbuffer);
- if (vacrel->aggressive)
- {
- if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
- break;
- }
- else
- {
- if ((vmskipflags & VISIBILITYMAP_ALL_VISIBLE) == 0)
- break;
- }
- vacuum_delay_point();
- next_unskippable_block++;
- }
- }
-
/*
- * We know we can't skip the current block. But set up
- * skipping_blocks to do the right thing at the following blocks.
+ * Can't skip this page safely. Must scan the page. But
+ * determine the next skippable range after the page first.
*/
- if (next_unskippable_block - blkno > SKIP_PAGES_THRESHOLD)
- skipping_blocks = true;
- else
- skipping_blocks = false;
+ all_visible_according_to_vm = all_visible_next_unskippable;
+ next_unskippable_block =
+ lazy_scan_skip_range(vacrel, &vmbuffer, blkno + 1,
+ &all_visible_next_unskippable,
+ &skipping_range);
- /*
- * Normally, the fact that we can't skip this block must mean that
- * it's not all-visible. But in an aggressive vacuum we know only
- * that it's not all-frozen, so it might still be all-visible.
- */
- if (vacrel->aggressive &&
- VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
- all_visible_according_to_vm = true;
+ Assert(next_unskippable_block >= blkno + 1);
}
else
{
- /*
- * The current page can be skipped if we've seen a long enough run
- * of skippable blocks to justify skipping it -- provided it's not
- * the last page in the relation (according to rel_pages).
- *
- * We always scan the table's last page to determine whether it
- * has tuples or not, even if it would otherwise be skipped. This
- * avoids having lazy_truncate_heap() take access-exclusive lock
- * on the table to attempt a truncation that just fails
- * immediately because there are tuples on the last page.
- */
- if (skipping_blocks && blkno < rel_pages - 1)
- {
- /*
- * Tricky, tricky. If this is in aggressive vacuum, the page
- * must have been all-frozen at the time we checked whether it
- * was skippable, but it might not be any more. We must be
- * careful to count it as a skipped all-frozen page in that
- * case, or else we'll think we can't update relfrozenxid and
- * relminmxid. If it's not an aggressive vacuum, we don't
- * know whether it was initially all-frozen, so we have to
- * recheck.
- */
- if (vacrel->aggressive ||
- VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
- vacrel->frozenskipped_pages++;
- continue;
- }
+ /* Last page always scanned (may need to set nonempty_pages) */
+ Assert(blkno < rel_pages - 1);
- /*
- * SKIP_PAGES_THRESHOLD (threshold for skipping) was not
- * crossed, or this is the last page. Scan the page, even
- * though it's all-visible (and possibly even all-frozen).
- */
+ if (skipping_range)
+ continue;
+
+ /* Not skipping pages from this range -- scan the page */
all_visible_according_to_vm = true;
}
- vacuum_delay_point();
-
- /*
- * We're not skipping this page using the visibility map, and so it is
- * (by definition) a scanned page. Any tuples from this page are now
- * guaranteed to be counted below, after some preparatory checks.
- */
vacrel->scanned_pages++;
+ /* Report as block scanned, update error traceback information */
+ pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
+ update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
+ blkno, InvalidOffsetNumber);
+
+ vacuum_delay_point();
+
/*
* Regularly check if wraparound failsafe should trigger.
*
@@ -1244,8 +1120,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
}
/*
- * Handle setting visibility map bit based on what the VM said about
- * the page before pruning started, and using prunestate
+ * Handle setting visibility map bit based on information from the VM
+ * (as of last lazy_scan_skip_range() call), and from prunestate
*/
if (!all_visible_according_to_vm && prunestate.all_visible)
{
@@ -1277,9 +1153,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/*
* As of PostgreSQL 9.2, the visibility map bit should never be set if
* the page-level bit is clear. However, it's possible that the bit
- * got cleared after we checked it and before we took the buffer
- * content lock, so we must recheck before jumping to the conclusion
- * that something bad has happened.
+ * got cleared after lazy_scan_skip_range() was called, so we must
+ * recheck with buffer lock before concluding that the VM is corrupt.
*/
else if (all_visible_according_to_vm && !PageIsAllVisible(page)
&& VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
@@ -1318,7 +1193,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
/*
* If the all-visible page is all-frozen but not marked as such yet,
* mark it as all-frozen. Note that all_frozen is only valid if
- * all_visible is true, so we must check both.
+ * all_visible is true, so we must check both prunestate fields.
*/
else if (all_visible_according_to_vm && prunestate.all_visible &&
prunestate.all_frozen &&
@@ -1424,6 +1299,114 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
Assert(!IsInParallelMode());
}
+/*
+ * lazy_scan_skip_range() -- set up range of skippable blocks using VM.
+ *
+ * lazy_scan_heap() calls here every time it needs to set up a new range of
+ * blocks to skip via the visibility map. Caller passes the next block in
+ * line. We return a next_unskippable_block for this range. When there are
+ * no skippable blocks we just return caller's next_block. The all-visible
+ * status of the returned block is set in *all_visible_next_unskippable for
+ * caller, too. Block usually won't be all-visible (since it's unskippable),
+ * but it can be during aggressive VACUUMs (as well as in certain edge cases).
+ *
+ * Sets *skipping_range to indicate if caller should actually skip this range.
+ * Costs and benefits drive our decision. Very small ranges won't be skipped.
+ *
+ * Note: our opinion of which blocks can be skipped can go stale immediately.
+ * It's okay if caller "misses" a page whose all-visible or all-frozen marking
+ * was concurrently cleared, though. All that matters is that caller scan all
+ * pages whose tuples might contain XIDs < OldestXmin, or XMIDs < OldestMxact.
+ * (Actually, non-aggressive VACUUMs can choose to skip all-visible pages with
+ * older XIDs/MXIDs. The vacrel->skippedallvis flag will be set here when the
+ * choice to skip such a range is actually made, making everything safe.)
+ */
+static BlockNumber
+lazy_scan_skip_range(LVRelState *vacrel, Buffer *vmbuffer,
+ BlockNumber next_block,
+ bool *all_visible_next_unskippable,
+ bool *skipping_range)
+{
+ BlockNumber rel_pages = vacrel->rel_pages,
+ next_unskippable_block = next_block,
+ nskippable_blocks = 0;
+ bool allvisinrange = false;
+
+ *all_visible_next_unskippable = true;
+ while (next_unskippable_block < rel_pages)
+ {
+ uint8 mapbits = visibilitymap_get_status(vacrel->rel,
+ next_unskippable_block,
+ vmbuffer);
+
+ if ((mapbits & VISIBILITYMAP_ALL_VISIBLE) == 0)
+ {
+ Assert((mapbits & VISIBILITYMAP_ALL_FROZEN) == 0);
+ *all_visible_next_unskippable = false;
+ break;
+ }
+
+ /*
+ * Caller must scan the last page to determine whether it has tuples
+ * (caller must have the opportunity to set vacrel->nonempty_pages).
+ * This rule avoids having lazy_truncate_heap() take access-exclusive
+ * lock on rel to attempt a truncation that fails anyway, just because
+ * there are tuples on the last page (it is likely that there will be
+ * tuples on other nearby pages as well, but those can be skipped).
+ *
+ * Implement this by always treating the last block as unsafe to skip.
+ */
+ if (next_unskippable_block == rel_pages - 1)
+ break;
+
+ /* DISABLE_PAGE_SKIPPING makes all skipping unsafe */
+ if (!vacrel->skipwithvm)
+ break;
+
+ /*
+ * Aggressive VACUUM caller can't skip pages just because they are
+ * all-visible. They may still skip all-frozen pages, which can't
+ * contain XIDs < OldestXmin (XIDs that aren't already frozen by now).
+ */
+ if ((mapbits & VISIBILITYMAP_ALL_FROZEN) == 0)
+ {
+ if (vacrel->aggressive)
+ break;
+
+ /*
+ * All-visible block is safe to skip in non-aggressive case. But
+ * remember that the final range contains such a block for later.
+ */
+ allvisinrange = true;
+ }
+
+ vacuum_delay_point();
+ next_unskippable_block++;
+ nskippable_blocks++;
+ }
+
+ /*
+ * We only skip a range with at least SKIP_PAGES_THRESHOLD consecutive
+ * pages. Since we're reading sequentially, the OS should be doing
+ * readahead for us, so there's no gain in skipping a page now and then.
+ * Skipping such a range might even discourage sequential detection.
+ *
+ * This test also enables more frequent relfrozenxid advancement during
+ * non-aggressive VACUUMs. If the range has any all-visible pages then
+ * skipping makes updating relfrozenxid unsafe, which is a real downside.
+ */
+ if (nskippable_blocks < SKIP_PAGES_THRESHOLD)
+ *skipping_range = false;
+ else
+ {
+ *skipping_range = true;
+ if (allvisinrange)
+ vacrel->skippedallvis = true;
+ }
+
+ return next_unskippable_block;
+}
+
/*
* lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
*
--
2.30.2
end of thread, other threads:[~2022-03-18 17:06 UTC | newest]
Thread overview: 90+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-29 07:00 BUG #17255: Server crashes in index_delete_sort_cmp() due to race condition with vacuum PG Bug reporting form <noreply@postgresql.org>
2021-10-29 14:55 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-10-30 21:39 ` Andres Freund <andres@anarazel.de>
2021-10-30 21:46 ` Peter Geoghegan <pg@bowt.ie>
2021-10-31 19:20 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-11-07 18:00 ` Alexander Lakhin <exclusion@gmail.com>
2021-11-08 17:28 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-11-08 18:32 ` Peter Geoghegan <pg@bowt.ie>
2021-11-09 15:02 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-11-09 17:51 ` Peter Geoghegan <pg@bowt.ie>
2021-11-09 23:31 ` Peter Geoghegan <pg@bowt.ie>
2021-11-10 03:07 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 03:08 ` Andres Freund <andres@anarazel.de>
2021-11-12 00:58 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 04:22 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 05:46 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 21:11 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 22:29 ` Andres Freund <andres@anarazel.de>
2021-11-12 22:46 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 22:57 ` Andres Freund <andres@anarazel.de>
2021-11-12 23:12 ` Peter Geoghegan <pg@bowt.ie>
2021-11-13 15:06 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-11-13 17:06 ` Peter Geoghegan <pg@bowt.ie>
2021-12-11 04:58 ` Andres Freund <andres@anarazel.de>
2021-12-13 12:21 ` Dmitry Dolgov <9erthalion6@gmail.com>
2021-12-13 18:48 ` Andres Freund <andres@anarazel.de>
2021-11-12 23:31 ` Andres Freund <andres@anarazel.de>
2021-11-12 23:47 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 23:48 ` Peter Geoghegan <pg@bowt.ie>
2021-11-12 23:56 ` Andres Freund <andres@anarazel.de>
2021-11-13 00:37 ` Peter Geoghegan <pg@bowt.ie>
2021-11-13 01:48 ` Andres Freund <andres@anarazel.de>
2021-11-13 01:57 ` Peter Geoghegan <pg@bowt.ie>
2021-11-13 02:32 ` Peter Geoghegan <pg@bowt.ie>
2021-11-16 02:39 ` Andres Freund <andres@anarazel.de>
2021-11-17 00:22 ` Peter Geoghegan <pg@bowt.ie>
2021-11-22 17:59 ` Andres Freund <andres@anarazel.de>
2021-11-22 20:41 ` Peter Geoghegan <pg@bowt.ie>
2021-11-22 22:34 ` Andres Freund <andres@anarazel.de>
2021-12-11 04:57 ` Andres Freund <andres@anarazel.de>
2021-12-11 22:03 ` Peter Geoghegan <pg@bowt.ie>
2021-12-11 22:51 ` Peter Geoghegan <pg@bowt.ie>
2022-01-10 19:31 ` Peter Geoghegan <pg@bowt.ie>
2022-01-12 19:25 ` Andres Freund <andres@anarazel.de>
2022-01-12 21:05 ` Peter Geoghegan <pg@bowt.ie>
2022-01-14 02:15 ` Andres Freund <andres@anarazel.de>
2022-01-14 17:00 ` Alexander Lakhin <exclusion@gmail.com>
2022-01-14 19:00 ` Andres Freund <andres@anarazel.de>
2022-02-03 20:54 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-02-04 04:19 ` Andres Freund <andres@anarazel.de>
2022-02-04 15:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-04 00:22 ` Peter Geoghegan <pg@bowt.ie>
2022-03-04 03:31 ` Peter Geoghegan <pg@bowt.ie>
2022-03-10 00:46 ` Andres Freund <andres@anarazel.de>
2022-03-10 00:25 ` Andres Freund <andres@anarazel.de>
2022-03-11 02:03 ` Peter Geoghegan <pg@bowt.ie>
2022-03-11 03:13 ` Andres Freund <andres@anarazel.de>
2022-03-11 03:40 ` Peter Geoghegan <pg@bowt.ie>
2022-03-11 03:46 ` Andres Freund <andres@anarazel.de>
2022-03-11 03:52 ` Peter Geoghegan <pg@bowt.ie>
2022-03-11 04:08 ` Andres Freund <andres@anarazel.de>
2022-03-11 04:55 ` Peter Geoghegan <pg@bowt.ie>
2022-03-12 21:49 ` Andres Freund <andres@anarazel.de>
2022-03-14 19:05 ` Peter Geoghegan <pg@bowt.ie>
2022-03-18 14:53 ` Andres Freund <andres@anarazel.de>
2022-03-18 17:06 ` Peter Geoghegan <pg@bowt.ie>
2022-03-11 18:42 ` Peter Geoghegan <pg@bowt.ie>
2021-11-10 19:20 ` Andres Freund <andres@anarazel.de>
2021-11-10 21:04 ` Peter Geoghegan <pg@bowt.ie>
2021-11-10 21:40 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 01:03 ` Andres Freund <andres@anarazel.de>
2021-11-11 00:47 ` Andres Freund <andres@anarazel.de>
2021-11-11 01:37 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 02:16 ` Andres Freund <andres@anarazel.de>
2021-11-11 02:20 ` Andres Freund <andres@anarazel.de>
2021-11-11 02:40 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 03:16 ` Andres Freund <andres@anarazel.de>
2021-11-11 03:33 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 20:05 ` Andres Freund <andres@anarazel.de>
2021-11-11 20:25 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 20:40 ` Peter Geoghegan <pg@bowt.ie>
2021-11-10 22:18 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 00:57 ` Andres Freund <andres@anarazel.de>
2021-11-11 01:19 ` Peter Geoghegan <pg@bowt.ie>
2021-11-11 01:43 ` Andres Freund <andres@anarazel.de>
2021-11-11 01:59 ` Peter Geoghegan <pg@bowt.ie>
2021-11-18 07:19 ` Andres Freund <andres@anarazel.de>
2021-11-19 00:36 ` Andres Freund <andres@anarazel.de>
2021-11-19 02:55 ` Peter Geoghegan <pg@bowt.ie>
2021-11-19 18:55 ` Andres Freund <andres@anarazel.de>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox