public inbox for [email protected]help / color / mirror / Atom feed
Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) 3+ messages / 3 participants [nested] [flat]
* Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) @ 2025-10-23 18:16 Devrim Gündüz <[email protected]> 2025-10-23 19:37 ` Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) David Rowley <[email protected]> 2025-10-23 20:24 ` RE: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) Scot Kreienkamp <[email protected]> 0 siblings, 2 replies; 3+ messages in thread From: Devrim Gündüz @ 2025-10-23 18:16 UTC (permalink / raw) To: [email protected]; Bala M <[email protected]> Hi, This happens because of the glibc version differrence between RHEL X and RHEL Y. At this point you either have to rebuild all indexes (sorry!) or redo the upgrade via logical replication (if it works for your app's behaviour) Devrim On 22 October 2025 12:03:06 EEST, Bala M <[email protected]> wrote: >Hi Team, > >We are facing an issue related to index corruption after migrating our >PostgreSQL 11 setup from *RHEL 7* to *RHEL 9* using *streaming replication* >(base backup method). > >After bringing up the standby on RHEL 9, we observed that certain tables >are not returning results when queries use indexed scans. Upon >investigation, the following check confirms index corruption: > >The same indexes work fine on the RHEL 7 (primary) side. However, on >the RHEL 9 replica, queries that rely on this index return zero >records. >Rebuilding the indexes fixed the issue temporarily but we have many >indexes and our DB size is more than 10TB. > >*Environment details:* > > - > > PostgreSQL Version: 11.15 > - > > OS on primary: RHEL 7.9 > - > > OS on standby: RHEL 9.6 > - > > Replication Type: Streaming replication (initialized using pg_basebackup) > - > > Data Directory initialized from RHEL 7 base backup > >*Issue Summary:* > > - > > Indexes appear and are the same size as per prod on standby after base > backup restore. > - > > We are able to read the data from the tables without index scans on > standy by RHEL 9. > - > > No filesystem or WAL errors observed in logs. > >Could this be related to OS-level binary or page layout differences between >RHEL 7 and RHEL 9 for PostgreSQL 11 binaries? >Any insights or recommended actions would be greatly appreciated. > > >Thanks & Regards, >*krishna.* ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) 2025-10-23 18:16 Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) Devrim Gündüz <[email protected]> @ 2025-10-23 19:37 ` David Rowley <[email protected]> 1 sibling, 0 replies; 3+ messages in thread From: David Rowley @ 2025-10-23 19:37 UTC (permalink / raw) To: Devrim Gündüz <[email protected]>; +Cc: [email protected]; Bala M <[email protected]> On Fri, 24 Oct 2025 at 07:17, Devrim Gündüz <[email protected]> wrote: > This happens because of the glibc version differrence between RHEL X and RHEL Y. At this point you either have to rebuild all indexes (sorry!) or redo the upgrade via logical replication (if it works for your app's behaviour) There was a mention in [1] about a wiki page which includes a query to identify which indexes could be affected. While that *could* be all indexes, it does seem unlikely. David [1] https://postgr.es/m/CAOg7f80JJCMHXhNLUjnHMwBFV-f9Msv4L=PML0CdaZ4n47uEeA@mail.gmail.com ^ permalink raw reply [nested|flat] 3+ messages in thread
* RE: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) 2025-10-23 18:16 Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) Devrim Gündüz <[email protected]> @ 2025-10-23 20:24 ` Scot Kreienkamp <[email protected]> 1 sibling, 0 replies; 3+ messages in thread From: Scot Kreienkamp @ 2025-10-23 20:24 UTC (permalink / raw) To: Devrim Gündüz <[email protected]>; [email protected] <[email protected]>; Bala M <[email protected]> Just wanted to clarify, this failed because OP used streaming which may not work between OS versions? But logical would have been just fine, between OS and PG versions or even to Windows? I always assumed streaming would “just work” as long as it’s the same major PG version and Linux-to-Linux regardless of OS/glibc version. That’s an awesome piece of design work BTW… congratulations and thank you for that. It never occurred to me that there could be an OS influencing factor like the glibc version for streaming replication. I almost took the same streaming path about two months ago to go from RH7 to RH8; instead opted to take some downtime overnight and do a dump/restore to go from RH7 to RH8 because logical replication was not an option and it seemed safer. Sounds like I chose the better path by chance. Thank you for sharing your knowledge. Scot Kreienkamp | Applications Infrastructure Architect | La-Z-Boy Corporate (734) 384-6403 | 1-734-915-1444 | [email protected] One La-Z-Boy Drive | Monroe, Michigan 48162 | la-z-boy.com<http://www.la-z-boy.com/; facebook.com/lazboy<http://facebook.com/lazboy; | instagram.com/lazboy<https://instagram.com/lazboy; | youtube.com/lazboy<http://youtube.com/lazboy; [cid:lazboy_2024_inc_navy_4a4d68ec-613a-4141-a2aa-d73a2ae749f6.png] From: Devrim Gündüz <[email protected]> Sent: Thursday, October 23, 2025 2:17 PM To: [email protected]; Bala M <[email protected]> Subject: Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) Hi, This happens because of the glibc version differrence between RHEL X and RHEL Y. At this point you either have to rebuild all indexes (sorry!) or redo the upgrade via logical replication (if it works for your app's behaviour) Devrim On 22 October 2025 12:03:06 EEST, Bala M <[email protected]<mailto:[email protected]>> wrote: Hi Team, We are facing an issue related to index corruption after migrating our PostgreSQL 11 setup from RHEL 7 to RHEL 9 using streaming replication (base backup method). After bringing up the standby on RHEL 9, we observed that certain tables are not returning results when queries use indexed scans. Upon investigation, the following check confirms index corruption: The same indexes work fine on the RHEL 7 (primary) side. However, on the RHEL 9 replica, queries that rely on this index return zero records. Rebuilding the indexes fixed the issue temporarily but we have many indexes and our DB size is more than 10TB. Environment details: · PostgreSQL Version: 11.15 · OS on primary: RHEL 7.9 · OS on standby: RHEL 9.6 · Replication Type: Streaming replication (initialized using pg_basebackup) · Data Directory initialized from RHEL 7 base backup Issue Summary: · Indexes appear and are the same size as per prod on standby after base backup restore. · We are able to read the data from the tables without index scans on standy by RHEL 9. · No filesystem or WAL errors observed in logs. Could this be related to OS-level binary or page layout differences between RHEL 7 and RHEL 9 for PostgreSQL 11 binaries? Any insights or recommended actions would be greatly appreciated. Thanks & Regards, krishna. This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you. Attachments: [image/png] lazboy_2024_inc_navy_4a4d68ec-613a-4141-a2aa-d73a2ae749f6.png (4.3K, 3-lazboy_2024_inc_navy_4a4d68ec-613a-4141-a2aa-d73a2ae749f6.png) download | view image ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2025-10-23 20:24 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-10-23 18:16 Re: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication) Devrim Gündüz <[email protected]> 2025-10-23 19:37 ` David Rowley <[email protected]> 2025-10-23 20:24 ` Scot Kreienkamp <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox