Message-ID: <01df0ca4-cf32-448f-a2cb-e44a8e9f7fd6@element.io>
Date: Fri, 4 Jul 2025 14:49:27 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Corrupt btree index includes rows that don't match
To: pgsql-general@lists.postgresql.org
References: <CAPo1J60Vcu+5G0EvvAZtYgTn6U6ADij3aVJ8WFVz77jP+Bd_Tw@mail.gmail.com>
From: Erik Johnston <erikj@element.io>
In-Reply-To: <CAPo1J60Vcu+5G0EvvAZtYgTn6U6ADij3aVJ8WFVz77jP+Bd_Tw@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------866EKVpxmTft7dDNXQA0xt9t"
Content-Language: en-GB
Archived-At: <https://www.postgresql.org/message-id/01df0ca4-cf32-448f-a2cb-e44a8e9f7fd6%40element.io>
Precedence: bulk

This is a multi-part message in MIME format.
--------------866EKVpxmTft7dDNXQA0xt9t
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi, a quick update:

- We have discovered that the corruption was present from before libicu=20
update.
- We ran `pg_amcheck --index state_groups_state_type_idx=20
--heapallindexed matrix`, which returned nothing
- We believe that means that (and matches what we see sampling) the=20
index has gained extra entries, i.e. that for a given state group it=20
does return all the relevant rows in the table /plus/ extra rows.

We are also seeing old state groups starting to point at rows that have=20
only just been inserted. For example, querying for 353864583 on the=20
primary it returns that row plus four rows that have been inserted=20
today, but on the backup from last week an index only scan for 353864583=20
only returns one row. This makes it feel like the corruption is ongoing?=20
Nothing should have modified that state group in the interim (they are=20
generally immutable).

This naively feels like when inserting a new row we sometimes add the=20
row to the index twice: once pointing from the correct state group to=20
the new row, and once from an old state group to the new row?


Thanks,
Erik

On 03/07/2025 18:07, Erik Johnston wrote:
>
> Hello,
>
>
> We're looking into a problem with our application and have tracked it=20
> down to index corruption, whereby we have many index rows pointing to=20
> the wrong tuples in the heap.
>
>
> Our table looks like:
>
>
> ```
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Table "matrix.state_groups_state=
"
> =C2=A0 =C2=A0Column =C2=A0 =C2=A0| =C2=A0Type =C2=A0| Collation | Nullabl=
e | Default
> -------------+--------+-----------+----------+---------
> =C2=A0state_group | bigint | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0|
> =C2=A0room_id =C2=A0 =C2=A0 | text =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> =C2=A0type =C2=A0 =C2=A0 =C2=A0 =C2=A0| text =C2=A0 | =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> =C2=A0state_key =C2=A0 | text =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> =C2=A0event_id =C2=A0 =C2=A0| text =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> Indexes:
> =C2=A0 =C2=A0 "state_groups_state_room_id_idx" brin (room_id) WITH=20
> (pages_per_range=3D'1')
> =C2=A0 =C2=A0 "state_groups_state_type_idx" btree (state_group, type,=20
> state_key), tablespace "postgres_second"
> Triggers:
> =C2=A0 =C2=A0 check_state_groups_state_deletion_trigger AFTER DELETE ON=
=20
> state_groups_state DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE=20
> FUNCTION check_state_groups_state_deletion()
>
> ```
>
>
> The symptoms we are noticing are that a DELETE or SELECT query=20
> includes rows that don't match the condition, as long as we issue a=20
> query that results in an Index Scan (not Index Only Scan):
>
>
> For example, including `ctid` in the query is enough to make the=20
> planner use an Index Scan:
>
>
> ```
>
> SELECT ctid, state_group FROM state_groups_state WHERE state_group =3D=20
> 483128098;
>
> =C2=A0 =C2=A0 =C2=A0 ctid =C2=A0 =C2=A0 =C2=A0| state_group
> ----------------+-------------
> =C2=A0(16669607,1) =C2=A0 | =C2=A0 483128098
> =C2=A0(424940858,20) | =C2=A0 963361875
> =C2=A0(16669606,53) =C2=A0| =C2=A0 483128098
> (3 rows)
>
> ```
>
>
> But with an Index Only Scan:
>
>
> ```
>
> SELECT state_group FROM state_groups_state WHERE state_group =3D 48312809=
8;
> =C2=A0state_group
> -------------
> =C2=A0 =C2=A0483128098
> =C2=A0 =C2=A0483128098
> =C2=A0 =C2=A0483128098
> (3 rows)
>
> ```
>
>
> Since including `ctid` in the SELECT columns causes the query to use=20
> an Index Scan (fetching tuples from the heap), this inconsistency=20
> leads us to believe that our index and heap disagree.
>
>
> Forcing a sequential scan with that same query only returns two rows=20
> matching that state group, which suggests that the index thinks there=20
> are more rows in the table than there actually are. (We do not believe=20
> anything can have deleted a row with state group 483128098). Also=20
> interestingly, querying (with the index re-enabled) for 963361875=20
> returns the same row as returned above, so the row is in the index twice.
>
>
> Another example state group (147961623) should only have a single row=20
> associated with it, and yet the index returns nearly 7000 rows=20
> (including the one we expect). The unexpected state groups are all in=20
> the range 794390760=E2=80=93794393085 (except one in 794411694), and also=
 have=20
> ctids in range (93454823,48) =E2=80=93 (93455621,49). The fact that these=
 are=20
> reasonably tight ranges feels suspicious. Note that the state group is=20
> a simple incrementing ID here.
>
>
> This table is quite large (about 6 TB) but we have sampled a few small=20
> ranges of it and found many instances of this type of corruption, in=20
> the first (approximate) half of the key range (0..561M out of 0..1034M).
>
>
> For historical reasons, the table and the index are on different=20
> tablespaces, but the same filesystem.
>
>
> We have sampled the table on our secondary server, and we see the same=20
> sort of corruption going on (though given the size of the data we=20
> don=E2=80=99t actually know if it's exactly the same).
>
>
> One coincidence is that we started seeing the first symptoms of this=20
> around the same time as libicu was updated with a security patch.=20
> However, postgres hasn=E2=80=99t been restarted and doesn=E2=80=99t refer=
ence the new=20
> version in its process maps. Plus state groups are integers anyway. We=20
> also use the C locale, not ICU.
>
>
> We=E2=80=99re currently running =E2=80=9Cpg_amcheck --index=20
> state_groups_state_type_idx --heapallindexed=E2=80=9D on our secondary to=
 see=20
> what it says, but we expect that to take a long time to complete.
>
>
> Thankfully, we have database backups so hopefullywe should be able to=20
> restore the data. However, any thoughts on how this happened or where=20
> to look next would be greatly appreciated. Thoughts on how to check=20
> our other indexes for corruption would also be very welcome.
>
>
> Thanks,
>
> Erik
>
>
>
> Further details of our setup:
>
>   * 2 servers in physical replication (one primary, one secondary as a
>     hot standby)
>       o both servers display the corruption
>   * ECC RAM
>   * 8 NVME SSD, raid10 (mdraid), LVM, ext4 filesystem.
>       o smartctl and mdadm report healthy disks
>   * Debian, postgres installed via apt.
>   * Postgres version: PostgreSQL 14.11 (Debian 14.11-1.pgdg120+1) on
>     x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bi=
t
>   * Kernel version 6.1.0-22-amd64, GLIBC 2.36-9+deb12u10
>
>
>
> Copyright =C2=A9 2025 Element - All rights reserved. The Element name, lo=
go=20
> and device are registered trademarks of New Vector Ltd. Registered=20
> number: 10873661. Registered in England and Wales. Registered address:=20
> 10 Queen Street Place, London, United Kingdom, EC4R 1AG.
>
> This message is intended for the addressee only and may contain=20
> private and confidential information or material which may be=20
> privileged. If this message has come to you in error please delete it=20
> immediately and do not copy it or show it to any other person.
>

--=20


Copyright =C2=A9 2025 Element - All rights reserved. The Element name, logo=
=20
and device are registered trademarks of New Vector Ltd. Registered number:=
=20
10873661. Registered in England and Wales. Registered address: 10 Queen=20
Street Place, London, United Kingdom, EC4R 1AG.

This message is intended=20
for the addressee only and may contain private and confidential information=
=20
or material which may be privileged. If this message has come to you in=20
error please delete it immediately and do not copy it or show it to any=20
other person.

--------------866EKVpxmTft7dDNXQA0xt9t
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8=
">
  </head>
  <body>
    <p>Hi, a quick update:</p>
    <p>- We have discovered that the corruption was present from before
      libicu update.<br>
      - We ran `pg_amcheck --index state_groups_state_type_idx
      --heapallindexed matrix`, which returned nothing<br>
      - We believe that means that (and matches what we see sampling)
      the index has gained extra entries, i.e. that for a given state
      group it does return all the relevant rows in the table <i>plus</i>
      extra rows.<br>
    </p>
    <p>We are also seeing old state groups starting to point at rows
      that have only just been inserted. For example, querying for
      353864583 on the primary it returns that row plus four rows that
      have been inserted today, but on the backup from last week an
      index only scan for 353864583 only returns one row. This makes it
      feel like the corruption is ongoing? Nothing should have modified
      that state group in the interim (they are generally immutable).</p>
    <p>This naively feels like when inserting a new row we sometimes add
      the row to the index twice: once pointing from the correct state
      group to the new row, and once from an old state group to the new
      row?<br>
    </p>
    <p><br>
    </p>
    Thanks,<br>
    Erik<br>
    <br>
    <p></p>
    <div class=3D"moz-cite-prefix">On 03/07/2025 18:07, Erik Johnston
      wrote:<br>
    </div>
    <blockquote type=3D"cite"
cite=3D"mid:CAPo1J60Vcu+5G0EvvAZtYgTn6U6ADij3aVJ8WFVz77jP+Bd_Tw@mail.gmail.=
com">
      <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DUTF=
-8">
      <div dir=3D"ltr">
        <div dir=3D"ltr"><span
id=3D"m_6451470738821132714gmail-docs-internal-guid-9eeedf76-7fff-9d92-7017=
-242b3674a090">
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Hello,</spa=
n></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">We're
                looking into a problem with our application and have
                tracked it down to index corruption, whereby we have
                many index rows pointing to the wrong tuples in the
                heap.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Our
                table looks like:</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
font
                face=3D"monospace">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0Table
                "matrix.state_groups_state"<br>
                =C2=A0 =C2=A0Column =C2=A0 =C2=A0| =C2=A0Type =C2=A0| Colla=
tion | Nullable | Default <br>
                -------------+--------+-----------+----------+---------<br>
                =C2=A0state_group | bigint | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| <br>
                =C2=A0room_id =C2=A0 =C2=A0 | text =C2=A0 | =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| <br>
                =C2=A0type =C2=A0 =C2=A0 =C2=A0 =C2=A0| text =C2=A0 | =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| <br>
                =C2=A0state_key =C2=A0 | text =C2=A0 | =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| <br>
                =C2=A0event_id =C2=A0 =C2=A0| text =C2=A0 | =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| <br>
                Indexes:<br>
                =C2=A0 =C2=A0 "state_groups_state_room_id_idx" brin (room_i=
d) WITH
                (pages_per_range=3D'1')<br>
                =C2=A0 =C2=A0 "state_groups_state_type_idx" btree (state_gr=
oup,
                type, state_key), tablespace "postgres_second"<br>
                Triggers:<br>
                =C2=A0 =C2=A0 check_state_groups_state_deletion_trigger AFT=
ER
                DELETE ON state_groups_state DEFERRABLE INITIALLY
                DEFERRED FOR EACH ROW EXECUTE FUNCTION
                check_state_groups_state_deletion()</font><br>
            </p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">The
                symptoms we are noticing are that a DELETE or SELECT
                query includes rows that don't match the condition, as
                long as we issue a query that results in an Index Scan
                (not Index Only Scan):</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">For
                example, including `ctid` in the query is enough to make
                the planner use an Index Scan:</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
font
                face=3D"monospace">SELECT ctid, state_group FROM
                state_groups_state WHERE state_group =3D 483128098;<br>
                <br>
                =C2=A0 =C2=A0 =C2=A0 ctid =C2=A0 =C2=A0 =C2=A0| state_group=
 <br>
                ----------------+-------------<br>
                =C2=A0(16669607,1) =C2=A0 | =C2=A0 483128098<br>
                =C2=A0(424940858,20) | =C2=A0 963361875<br>
                =C2=A0(16669606,53) =C2=A0| =C2=A0 483128098<br>
                (3 rows)</font></p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">But
                with an Index Only Scan:</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
font
                face=3D"monospace">SELECT state_group FROM
                state_groups_state WHERE state_group =3D 483128098;<br>
                =C2=A0state_group <br>
                -------------<br>
                =C2=A0 =C2=A0483128098<br>
                =C2=A0 =C2=A0483128098<br>
                =C2=A0 =C2=A0483128098<br>
                (3 rows)</font><br>
            </p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">```</span><=
/p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Since
                including `ctid` in the SELECT columns causes the query
                to use an Index Scan (fetching tuples from the heap),
                this inconsistency leads us to believe that our index
                and heap disagree.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Forcing
                a sequential scan with that same query only returns two
                rows matching that state group, which suggests that the
                index thinks there are more rows in the table than there
                actually are. (We do not believe anything can have
                deleted a row with state group 483128098). Also
                interestingly, querying (with the index re-enabled) for
                963361875 returns the same row as returned above, so the
                row is in the index twice.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Another
                example state group (147961623) should only have a
                single row associated with it, and yet the index returns
                nearly 7000 rows (including the one we expect). The
                unexpected state groups are all in the range
                794390760=E2=80=93794393085 (except one in 794411694), and =
also
                have ctids in range (93454823,48) =E2=80=93 (93455621,49). =
The
                fact that these are reasonably tight ranges feels
                suspicious. Note that the state group is a simple
                incrementing ID here.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">This
                table is quite large (about 6 TB) but we have sampled a
                few small ranges of it and found many instances of this
                type of corruption, in the first (approximate) half of
                the key range (0..561M out of 0..1034M).</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">For
                historical reasons, the table and the index are on
                different tablespaces, but the same filesystem.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">We
                have sampled the table on our secondary server, and we
                see the same sort of corruption going on (though given
                the size of the data we don=E2=80=99t actually know if it's
                exactly the same).</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">One
                coincidence is that we started seeing the first symptoms
                of this around the same time as libicu was updated with
                a security patch. However, postgres hasn=E2=80=99t been
                restarted and doesn=E2=80=99t reference the new version in =
its
                process maps. Plus state groups are integers anyway. We
                also use the C locale, not ICU.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">We=E2=80=99=
re
                currently running =E2=80=9Cpg_amcheck --index
                state_groups_state_type_idx --heapallindexed=E2=80=9D on ou=
r
                secondary to see what it says, but we expect that to
                take a long time to complete.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Thankfully,
                we have database backups so </span><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-style:italic;font-variant-numeric:normal;font-=
variant-east-asian:normal;font-variant-alternates:normal;vertical-align:bas=
eline">hopefully</span><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">
                we should be able to restore the data. However, any
                thoughts on how this happened or where to look next
                would be greatly appreciated. Thoughts on how to check
                our other indexes for corruption would also be very
                welcome.</span></p>
            <br>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Thanks,</sp=
an></p>
            <p style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt">=
<span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Erik</span>=
</p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline"><br>
              </span></p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline"><br>
              </span></p>
            <p dir=3D"ltr"
              style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><=
span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Further
                details of our setup:<br>
              </span></p>
            <ul>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">2
                  servers in physical replication (one primary, one
                  secondary as a hot standby)</span></li>
              <ul>
                <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">both
                    servers display the corruption</span></li>
              </ul>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">ECC
                  RAM</span></li>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">8
                  NVME SSD, raid10 (mdraid), LVM, ext4 filesystem.</span></=
li>
              <ul>
                <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">smartctl
                    and mdadm report healthy disks</span></li>
              </ul>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Debian,
                  postgres installed via apt.</span></li>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Postgres
                  version: PostgreSQL 14.11 (Debian 14.11-1.pgdg120+1)
                  on x86_64-pc-linux-gnu, compiled by gcc (Debian
                  12.2.0-14) 12.2.0, 64-bit</span></li>
              <li><span
style=3D"font-size:11pt;font-family:Arial,sans-serif;color:rgb(0,0,0);backg=
round-color:transparent;font-variant-numeric:normal;font-variant-east-asian=
:normal;font-variant-alternates:normal;vertical-align:baseline">Kernel
                  version 6.1.0-22-amd64, GLIBC 2.36-9+deb12u10</span></li>
            </ul>
          </span>
          <p dir=3D"ltr"
            style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br=
>
          </p>
        </div>
      </div>
      <br>
      <p dir=3D"ltr"><span style=3D"color:gray"><font size=3D"1"><span
              style=3D"font-family:Verdana"><span
style=3D"font-weight:400;font-style:normal;text-decoration:none">Copyright
                =C2=A9 2025 Element - All rights reserved. The Element name=
,
                logo and device are registered trademarks of New Vector
                Ltd. Registered number: 10873661. Registered in England
                and Wales. Registered address: 10 Queen Street Place,
                London, United Kingdom, EC4R 1AG.</span></span></font></spa=
n></p>
      <p dir=3D"ltr"><span style=3D"color:gray"><font size=3D"1"><span
              style=3D"font-family:Verdana"><span
style=3D"font-weight:400;font-style:normal;text-decoration:none">This
                message is intended for the addressee only and may
                contain private and confidential information or material
                which may be privileged. If this message has come to you
                in error please delete it immediately and do not copy it
                or show it to any other person.</span></span></font></span>=
</p>
    </blockquote>
    <br>
  </body>
</html>

<br>
<p dir=3D"ltr"><span style=3D"color:gray"><font size=3D"1"><span style=3D"f=
ont-family:Verdana"><span style=3D"font-weight:400;font-style:normal;text-d=
ecoration:none">Copyright =C2=A9 2025 Element - All rights reserved. The El=
ement name, logo and device are registered trademarks of New Vector Ltd. Re=
gistered number: 10873661. Registered in England and Wales. Registered addr=
ess: 10 Queen Street Place, London, United Kingdom, EC4R 1AG.</span></span>=
</font></span></p><p dir=3D"ltr"><span style=3D"color:gray"><font size=3D"1=
"><span style=3D"font-family:Verdana"><span style=3D"font-weight:400;font-s=
tyle:normal;text-decoration:none">This message is intended for the addresse=
e only and may contain private and confidential information or material whi=
ch may be privileged. If this message has come to you in error please delet=
e it immediately and do not copy it or show it to any other person.</span><=
/span></font></span></p>
--------------866EKVpxmTft7dDNXQA0xt9t--