Subject: Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication)

From: Devrim Gündüz <[email-masked]>
Date: Thu, 23 Oct 2025 21:16:59 +0300

Hi,

This happens because of the glibc version differrence between RHEL X and RHEL Y. At this point you either have to rebuild all indexes (sorry!) or redo the upgrade via logical replication (if it works for your app's behaviour)

Devrim

On 22 October 2025 12:03:06 EEST, Bala M <[email-masked]> wrote:
>Hi Team,
>
>We are facing an issue related to index corruption after migrating our
>PostgreSQL 11 setup from *RHEL 7* to *RHEL 9* using *streaming replication*
>(base backup method).
>
>After bringing up the standby on RHEL 9, we observed that certain tables
>are not returning results when queries use indexed scans. Upon
>investigation, the following check confirms index corruption:
>
>The same indexes work fine on the RHEL 7 (primary) side. However, on
>the RHEL 9 replica, queries that rely on this index return zero
>records.
>Rebuilding the indexes  fixed the issue temporarily but we have many
>indexes  and our DB size is more than 10TB.
>
>*Environment details:*
>
>   -
>
>   PostgreSQL Version: 11.15
>   -
>
>   OS on primary: RHEL 7.9
>   -
>
>   OS on standby: RHEL 9.6
>   -
>
>   Replication Type: Streaming replication (initialized using pg_basebackup)
>   -
>
>   Data Directory initialized from RHEL 7 base backup
>
>*Issue Summary:*
>
>   -
>
>   Indexes appear and are the same size as per prod  on standby after base
>   backup restore.
>   -
>
>   We are able to read  the data from the tables without index scans on
>   standy by RHEL 9.
>   -
>
>   No filesystem or WAL errors observed in logs.
>
>Could this be related to OS-level binary or page layout differences between
>RHEL 7 and RHEL 9 for PostgreSQL 11 binaries?
>Any insights or recommended actions would be greatly appreciated.
>
>
>Thanks & Regards,
>*krishna.*

---

From: David Rowley <[email-masked]>
Date: Fri, 24 Oct 2025 08:37:17 +1300

> This happens because of the glibc version differrence between RHEL X and RHEL Y. At this point you either have to rebuild all indexes (sorry!) or redo the upgrade via logical replication (if it wor…

There was a mention in [1] about a wiki page which includes a query to
identify which indexes could be affected. While that *could* be all
indexes, it does seem unlikely.

David

[1] https://postgr.es/m/CAOg7f80JJCMHXhNLUjnHMwBFV-f9Msv4L=[email-masked]

---

From: Scot Kreienkamp <[email-masked]>
Date: Thu, 23 Oct 2025 20:24:08 +0000

Just wanted to clarify, this failed because OP used streaming which may not work between OS versions?  But logical would have been just fine, between OS and PG versions or even to Windows?  I always assumed streaming would “just work” as long as it’s the same major PG version and Linux-to-Linux regardless of OS/glibc version.  That’s an awesome piece of design work BTW… congratulations and thank you for that.  It never occurred to me that there could be an OS influencing factor like the glibc version for streaming replication.

I almost took the same streaming path about two months ago to go from RH7 to RH8; instead opted to take some downtime overnight and do a dump/restore to go from RH7 to RH8 because logical replication was not an option and it seemed safer.  Sounds like I chose the better path by chance.

Thank you for sharing your knowledge.

Scot Kreienkamp | Applications Infrastructure Architect | La-Z-Boy Corporate
(734) 384-6403 | 1-734-915-1444 | [email-masked]
One La-Z-Boy Drive | Monroe, Michigan 48162 | la-z-boy.com<http://www.la-z-boy.com/>
 facebook.com/lazboy<http://facebook.com/lazboy>  | instagram.com/lazboy<https://instagram.com/lazboy> | youtube.com/lazboy<http://youtube.com/lazboy>

From: Devrim Gündüz <[email-masked]>
Sent: Thursday, October 23, 2025 2:17 PM
To: [email-masked]; Bala M <[email-masked]>
Subject: Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication)

Hi,

Devrim

On 22 October 2025 12:03:06 EEST, Bala M <[email-masked]<mailto:[email-masked]>> wrote:

Hi Team,

We are facing an issue related to index corruption after migrating our PostgreSQL 11 setup from RHEL 7 to RHEL 9 using streaming replication (base backup method).

After bringing up the standby on RHEL 9, we observed that certain tables are not returning results when queries use indexed scans. Upon investigation, the following check confirms index corruption:

The same indexes work fine on the RHEL 7 (primary) side. However, on the RHEL 9 replica, queries that rely on this index return zero records.

Rebuilding the indexes  fixed the issue temporarily but we have many indexes  and our DB size is more than 10TB.

Environment details:

· PostgreSQL Version: 11.15

· OS on primary: RHEL 7.9

· OS on standby: RHEL 9.6

· Replication Type: Streaming replication (initialized using pg_basebackup)

· Data Directory initialized from RHEL 7 base backup

Issue Summary:

· Indexes appear and are the same size as per prod  on standby after base backup restore.

· We are able to read  the data from the tables without index scans on standy by RHEL 9.

· No filesystem or WAL errors observed in logs.

Could this be related to OS-level binary or page layout differences between RHEL 7 and RHEL 9 for PostgreSQL 11 binaries?
Any insights or recommended actions would be greatly appreciated.

Thanks & Regards,
krishna.

This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.