From: Greg Burd <greg@burd.me>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Tepid: selective index updates for heap relations
Date: Mon, 13 Jul 2026 17:06:49 -0400
Message-ID: <efe30c88-cf01-4c48-b76e-777547babcea@app.fastmail.com> (raw)
In-Reply-To: <ed69804a-aa09-489b-af3f-6c804b2efd94@app.fastmail.com>
References: <af02b486-6adc-494c-a357-1dd72f655dcf@app.fastmail.com>
<CAPpHfdvOLrULXQn+s+bEG4QiVrps+_3mg2EjUN39OYUKoBE20w@mail.gmail.com>
<97d5e5a3-f8e2-49ec-bd52-5b43e5d2b30a@app.fastmail.com>
<b078fb42-cecb-4868-8a65-11f9ce484868@app.fastmail.com>
<3dc5d57e-6c77-4713-b97e-bf0a5a891226@app.fastmail.com>
<ed69804a-aa09-489b-af3f-6c804b2efd94@app.fastmail.com>
Hello,
Attached is v64, rebased onto current master (3ed29c42a40), this time as
a .tar.gz attachment to avoid getting moderated. What changed vs v63
(posted hours ago... apologies for the churn)?
Three patches changed in content; the other seven differ only by rebase-
induced hunk offsets. All three changes address findings from an automated
review of v63, each verified against the code (reproduced before accepting)
rather than taken on faith. Two were real and are fixed; one was declined
with reasoning below.
0005: BitmapAnd/BitmapOr false negative -- amcheck false positive fixed
bt_index_check(..., heapallindexed => true) could raise a spurious "heap
tuple ... lacks matching index tuple" on an index maintained by HOT-indexed
(SIU) updates, when nbtree deduplication had merged the fresh entries into a
posting list.
A HOT-indexed fresh entry is inserted with the may-be-stale marker
(ItemPointerSIUMaybeStaleFlag, bit 14 of the offset) set on its heap TID.
verify_nbtree already strips that marker before fingerprinting a plain leaf
tuple, so its bloom image matches the unflagged heap TID from the heap scan.
But when many such entries share one key, deduplication copies the flagged
heap TIDs verbatim into a posting list's heap-TID array, and the posting
branch fingerprinted each element without stripping -- so a flagged element
never matched the plain heap TID and heapallindexed reported a false
positive. (My earlier belief that only plain leaf tuples carry the marker
was wrong; the fix and a corrected comment reflect that.)
Fix: strip the marker per element on the palloc'd plain tuple that
bt_posting_plain_tuple() produces, mirroring the existing plain-tuple strip.
Deterministic reproducer (fails before, passes after): a two-index table,
fillfactor=10, all rows UPDATEd to a single key value twice so the SIU fresh
entries dedup into one posting list of flagged live TIDs, then
bt_index_check(heapallindexed => true). Added as a regression test in 0008.
0007: amcheck HOT-indexed chain recognition -- over-broad exemption tightened
verify_heapam's "HOT chains should not intersect" check exempts the case
where several redirects legitimately converge on one live tuple (a collapsed
SIU chain: the root and each entry-bearing dead member become a redirect to
the survivor). v63 keyed that exemption on the target merely carrying
HEAP_INDEXED_UPDATED, which is set on any SIU-produced HOT tuple -- so a
genuinely corrupt page where two unrelated redirects converge on such a
tuple would have been silently accepted, defeating exactly the check amcheck
exists to make.
The fix was to exempt only the actual collapse shape -- the target is a
heap-only HOT-indexed survivor (heap-only was already verified) AND the
other predecessor forwarding to it is itself a redirect or a collapse
stub. An ordinary tuple converging there, or a non-HOT-indexed target, is
still reported. Verified this does not regress legitimate collapse: a
multi-redirect-to-survivor page still passes verify_heapam cleanly.
0008: statistics + comprehensive test suite
- Adds the posting-list-dedup heapallindexed regression test for the 0005
fix above (section 34).
best.
-greg
Attachments:
[application/gzip] v64-tepid.tar.gz (200.8K, ../efe30c88-cf01-4c48-b76e-777547babcea@app.fastmail.com/2-v64-tepid.tar.gz)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@burd.me
Subject: Re: Tepid: selective index updates for heap relations
In-Reply-To: <efe30c88-cf01-4c48-b76e-777547babcea@app.fastmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox