public inbox for [email protected]
help / color / mirror / Atom feedFrom: Erik Johnston <[email protected]>
To: [email protected]
Subject: Re: Corrupt btree index includes rows that don't match
Date: Fri, 4 Jul 2025 15:59:55 +0100
Message-ID: <CAPo1J60aoiwv=1HViWSiFEx9vpoj69cOfujCJRhy4Zjyw+QFaw@mail.gmail.com> (raw)
In-Reply-To: <CANzqJaAW2kwAyeJzboUCsbCpCk9ERDF1aZrW9OEa+DSqpPQxjA@mail.gmail.com>
References: <CAPo1J60Vcu+5G0EvvAZtYgTn6U6ADij3aVJ8WFVz77jP+Bd_Tw@mail.gmail.com>
<[email protected]>
<CANzqJaAW2kwAyeJzboUCsbCpCk9ERDF1aZrW9OEa+DSqpPQxjA@mail.gmail.com>
On Fri, 4 Jul 2025, 15:38 Ron Johnson, <[email protected]> wrote:
> On Fri, Jul 4, 2025 at 9:49 AM Erik Johnston <[email protected]> wrote:
>
>> Hi, a quick update:
>>
>> - We have discovered that the corruption was present from before libicu
>> update.
>> - We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed
>> matrix`, which returned nothing
>> - We believe that means that (and matches what we see sampling) the index
>> has gained extra entries, i.e. that for a given state group it does return
>> all the relevant rows in the table *plus* extra rows.
>>
>> We are also seeing old state groups starting to point at rows that have
>> only just been inserted. For example, querying for 353864583 on the primary
>> it returns that row plus four rows that have been inserted today, but on
>> the backup from last week an index only scan for 353864583 only returns one
>> row. This makes it feel like the corruption is ongoing? Nothing should have
>> modified that state group in the interim (they are generally immutable).
>>
>> This naively feels like when inserting a new row we sometimes add the row
>> to the index twice: once pointing from the correct state group to the new
>> row, and once from an old state group to the new row?
>>
>>
> Are checksums enabled in the instance?
>
Alas not.
We've also now found that the index on the backup does in fact point to
those ctids after all, but they are marked as dead. So at some point
between then and when we inserted the new row at that ctid today those
entries were marked undead.
--
Copyright © 2025 Element - All rights reserved. The Element name, logo
and device are registered trademarks of New Vector Ltd. Registered number:
10873661. Registered in England and Wales. Registered address: 10 Queen
Street Place, London, United Kingdom, EC4R 1AG.
This message is intended
for the addressee only and may contain private and confidential information
or material which may be privileged. If this message has come to you in
error please delete it immediately and do not copy it or show it to any
other person.
view thread (6+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: Corrupt btree index includes rows that don't match
In-Reply-To: <CAPo1J60aoiwv=1HViWSiFEx9vpoj69cOfujCJRhy4Zjyw+QFaw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox