MIME-Version: 1.0
From: Alessandro Regolini <alessandro@regolini.it>
Date: Wed, 1 Jul 2026 10:12:13 +0200
Message-ID: 
 <CAFDALapqu1NLj-svUV-sPdLs-e0DMEiaoS-kTFXhUqtjqn7M=A@mail.gmail.com>
Subject: BTree index corruption (heap-tid-past-end, unexpected zero page,
 misplaced TID in posting list) recurring on high-churn tables, PG 18.3,
 data_checksums=on, no preceding crash
To: pgsql-bugs@lists.postgresql.org
Content-Type: multipart/alternative; boundary="00000000000060274706558840b4"
Archived-At: 
 <https://www.postgresql.org/message-id/CAFDALapqu1NLj-svUV-sPdLs-e0DMEiaoS-kTFXhUqtjqn7M%3DA%40mail.gmail.com>
Precedence: bulk

--00000000000060274706558840b4
Content-Type: text/plain; charset="UTF-8"

PostgreSQL version:
PostgreSQL 18.3 (Ubuntu 18.3-1.pgdg24.04+1) on x86_64-pc-linux-gnu,
compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
OS: Ubuntu 24.04 (noble), running as a KVM/QEMU guest
data_checksums: on
shared_buffers: 128MB
btree deduplication: enabled (default)
Storage: ext4 (rw,relatime,errors=remount-ro) on a dedicated virtual disk
Relevant GUCs: fsync=on, full_page_writes=on, wal_level=replica,
autovacuum=on


*Summary*-------
Over ~4 weeks we hit recurring btree index corruption on multiple
independent
databases hosted on the same physical host (~80 PG 18 clusters, one DB per
cluster). Corruption concentrates exclusively on high-churn tables
(INSERT + DELETE + HOT update + frequent autovacuum: application cache and a
message-queue table). We observed three distinct signatures, all of them
btree-vs-heap desync:

  1. ERROR: heap tid from index tuple (X,Y) points past end of heap page
     line pointer array at offset NNN of block B in index "<idx>"
  2. index "<idx>" contains unexpected zero page at block B
  3. posting list contains misplaced TID in index "<idx>"   (btree dedup)

Timeline (verbatim runtime + amcheck log lines, times CEST)
-----------------------------------------------------------

*DB "soave":*  2026-06-05 .. 2026-06-08  ERROR: heap tid from index tuple
points past end
    of heap page line pointer array at offset 272 of block N in index
    "django_webix_sender_messagerecipient_pkey"
    -> emitted every 4h by a periodic query, then stopped on its own.
  2026-06-30 09:35 / 09:46  bt_index_check(heapallindexed=>true) reported:
    - index "qdc_magazzino_articolo_id_prev_a0195938"
      contains unexpected zero page at block N
    - index "django_webix_sender_messagerecipient_content_type_id_7552397e"
      posting list contains misplaced TID


*DB "ermes":*  2026-06-24 .. 2026-06-30  ERROR: heap tid from index tuple
points past end
    of heap page line pointer array at offset 248 of block N in index
    "mpaapp_cache_content_type_id_object_i_f6ecfefa_uniq"
    -> dozens of occurrences over the window.

Key facts
---------
- data_checksums=on and NO invalid-page / checksum-failure error was ever
  logged. The corrupt pages therefore carry valid checksums but structurally
  wrong content: they were written by PostgreSQL, this is not a torn write
or
  a storage-layer bit rot detected by checksums.
- No crash, PANIC, OOM or immediate shutdown preceded any event. The
postmaster
  was up continuously across each window. Host kernel journal shows zero
MCE /
  ECC / OOM events in the last 90 days.
- REINDEX INDEX CONCURRENTLY fixed every case with zero data loss (the heap
was
  always intact and fully readable), which points to an index-vs-heap desync
  rather than heap damage.
- The affected indexes are the busiest tables (application cache and message
  queue), constantly vacuumed. Corruption is intermittent and self-heals
after
  VACUUM / REINDEX.
- The same signature family reproduced on independent databases with
identical
  schema (same application deployed many times), which argues against random
  hardware/memory corruption: a random flip would not preferentially target
the
  same vacuum-heavy indexes with the same error text across separate
clusters.
- This is not a collation/glibc issue: the symptoms are physical structural
  desync (heap-tid-past-end, zero page, misplaced TID), not wrong sort
order.
- fsync=on and full_page_writes=on, so the two most common self-inflicted
  corruption causes are excluded.


*What we can provide on the next occurrence*
------------------------------------------
We run a periodic amcheck sweep, so we usually catch a fresh case before
reindexing. Before REINDEX we can capture, for the affected block:
  - contrib/pageinspect: bt_page_items() of the index block and
    heap_page_items() / page_header() of the referenced heap block
  - full bt_index_check / bt_index_parent_check output
  - verify_heapam() of the underlying table
Please tell us which dumps would be most useful to root-cause this (btree
deduplication / VACUUM interaction is our current suspicion).

# Alessandro Regolini

--00000000000060274706558840b4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>PostgreSQL version:<br>PostgreSQL 18.3 (Ubuntu 18.3-1=
.pgdg24.04+1) on x86_64-pc-linux-gnu,<br>compiled by gcc (Ubuntu 13.3.0-6ub=
untu2~24.04.1) 13.3.0, 64-bit<br>OS: Ubuntu 24.04 (noble), running as a KVM=
/QEMU guest<br>data_checksums: on<br>shared_buffers: 128MB<br>btree dedupli=
cation: enabled (default)<br>Storage: ext4 (rw,relatime,errors=3Dremount-ro=
) on a dedicated virtual disk<br>Relevant GUCs: fsync=3Don, full_page_write=
s=3Don, wal_level=3Dreplica, autovacuum=3Don<br><br><b>Summary<br></b>-----=
--<br>Over ~4 weeks we hit recurring btree index corruption on multiple ind=
ependent<br>databases hosted on the same physical host (~80 PG 18 clusters,=
 one DB per<br>cluster). Corruption concentrates exclusively on high-churn =
tables<br>(INSERT + DELETE + HOT update + frequent autovacuum: application =
cache and a<br>message-queue table). We observed three distinct signatures,=
 all of them<br>btree-vs-heap desync:<br><br>=C2=A0 1. ERROR: heap tid from=
 index tuple (X,Y) points past end of heap page<br>=C2=A0 =C2=A0 =C2=A0line=
 pointer array at offset NNN of block B in index &quot;&lt;idx&gt;&quot;<br=
>=C2=A0 2. index &quot;&lt;idx&gt;&quot; contains unexpected zero page at b=
lock B<br>=C2=A0 3. posting list contains misplaced TID in index &quot;&lt;=
idx&gt;&quot; =C2=A0 (btree dedup)<br><br>Timeline (verbatim runtime + amch=
eck log lines, times CEST)<br>---------------------------------------------=
--------------<br><b>DB &quot;soave&quot;:<br></b>=C2=A0 2026-06-05 .. 2026=
-06-08 =C2=A0ERROR: heap tid from index tuple points past end<br>=C2=A0 =C2=
=A0 of heap page line pointer array at offset 272 of block N in index<br>=
=C2=A0 =C2=A0 &quot;django_webix_sender_messagerecipient_pkey&quot;<br>=C2=
=A0 =C2=A0 -&gt; emitted every 4h by a periodic query, then stopped on its =
own.<br>=C2=A0 2026-06-30 09:35 / 09:46 =C2=A0bt_index_check(heapallindexed=
=3D&gt;true) reported:<br>=C2=A0 =C2=A0 - index &quot;qdc_magazzino_articol=
o_id_prev_a0195938&quot;<br>=C2=A0 =C2=A0 =C2=A0 contains unexpected zero p=
age at block N<br>=C2=A0 =C2=A0 - index &quot;django_webix_sender_messagere=
cipient_content_type_id_7552397e&quot;<br>=C2=A0 =C2=A0 =C2=A0 posting list=
 contains misplaced TID<br><br><b>DB &quot;ermes&quot;:<br></b>=C2=A0 2026-=
06-24 .. 2026-06-30 =C2=A0ERROR: heap tid from index tuple points past end<=
br>=C2=A0 =C2=A0 of heap page line pointer array at offset 248 of block N i=
n index<br>=C2=A0 =C2=A0 &quot;mpaapp_cache_content_type_id_object_i_f6ecfe=
fa_uniq&quot;<br>=C2=A0 =C2=A0 -&gt; dozens of occurrences over the window.=
<br><br>Key facts<br>---------<br>- data_checksums=3Don and NO invalid-page=
 / checksum-failure error was ever<br>=C2=A0 logged. The corrupt pages ther=
efore carry valid checksums but structurally<br>=C2=A0 wrong content: they =
were written by PostgreSQL, this is not a torn write or<br>=C2=A0 a storage=
-layer bit rot detected by checksums.<br>- No crash, PANIC, OOM or immediat=
e shutdown preceded any event. The postmaster<br>=C2=A0 was up continuously=
 across each window. Host kernel journal shows zero MCE /<br>=C2=A0 ECC / O=
OM events in the last 90 days.<br>- REINDEX INDEX CONCURRENTLY fixed every =
case with zero data loss (the heap was<br>=C2=A0 always intact and fully re=
adable), which points to an index-vs-heap desync<br>=C2=A0 rather than heap=
 damage.<br>- The affected indexes are the busiest tables (application cach=
e and message<br>=C2=A0 queue), constantly vacuumed. Corruption is intermit=
tent and self-heals after<br>=C2=A0 VACUUM / REINDEX.<br>- The same signatu=
re family reproduced on independent databases with identical<br>=C2=A0 sche=
ma (same application deployed many times), which argues against random<br>=
=C2=A0 hardware/memory corruption: a random flip would not preferentially t=
arget the<br>=C2=A0 same vacuum-heavy indexes with the same error text acro=
ss separate clusters.<br>- This is not a collation/glibc issue: the symptom=
s are physical structural<br>=C2=A0 desync (heap-tid-past-end, zero page, m=
isplaced TID), not wrong sort order.<br>- fsync=3Don and full_page_writes=
=3Don, so the two most common self-inflicted<br>=C2=A0 corruption causes ar=
e excluded.<br><br><b>What we can provide on the next occurrence<br></b>---=
---------------------------------------<br>We run a periodic amcheck sweep,=
 so we usually catch a fresh case before<br>reindexing. Before REINDEX we c=
an capture, for the affected block:<br>=C2=A0 - contrib/pageinspect: bt_pag=
e_items() of the index block and<br>=C2=A0 =C2=A0 heap_page_items() / page_=
header() of the referenced heap block<br>=C2=A0 - full bt_index_check / bt_=
index_parent_check output<br>=C2=A0 - verify_heapam() of the underlying tab=
le<br>Please tell us which dumps would be most useful to root-cause this (b=
tree<br>deduplication / VACUUM interaction is our current suspicion).<br></=
div><div><br></div><div><div dir=3D"ltr" class=3D"gmail_signature" data-sma=
rtmail=3D"gmail_signature"><div dir=3D"ltr"># Alessandro Regolini</div></di=
v></div></div>

--00000000000060274706558840b4--