Subject: Index corruption issue after migration from RHEL 7 to RHEL 9 (PostgreSQL 11 streaming replication)

From: Bala M <[email-masked]>
Date: Wed, 22 Oct 2025 14:33:06 +0530

Hi Team,

We are facing an issue related to index corruption after migrating our
PostgreSQL 11 setup from *RHEL 7* to *RHEL 9* using *streaming replication*
(base backup method).

After bringing up the standby on RHEL 9, we observed that certain tables
are not returning results when queries use indexed scans. Upon
investigation, the following check confirms index corruption:

The same indexes work fine on the RHEL 7 (primary) side. However, on
the RHEL 9 replica, queries that rely on this index return zero
records.
Rebuilding the indexes  fixed the issue temporarily but we have many
indexes  and our DB size is more than 10TB.

*Environment details:*

   -

   PostgreSQL Version: 11.15
   -

   OS on primary: RHEL 7.9
   -

   OS on standby: RHEL 9.6
   -

   Replication Type: Streaming replication (initialized using pg_basebackup)
   -

   Data Directory initialized from RHEL 7 base backup

*Issue Summary:*

   -

   Indexes appear and are the same size as per prod  on standby after base
   backup restore.
   -

   We are able to read  the data from the tables without index scans on
   standy by RHEL 9.
   -

   No filesystem or WAL errors observed in logs.

Could this be related to OS-level binary or page layout differences between
RHEL 7 and RHEL 9 for PostgreSQL 11 binaries?
Any insights or recommended actions would be greatly appreciated.

Thanks & Regards,
*krishna.*

---

From: Chris Wilson <[email-masked]>
Date: Wed, 22 Oct 2025 13:30:12 +0100

Hi Krishna,

This might be due to the upgrade in glibc between RHEL7 and RHEL8, which
includes updated collations that change sort orders and thus invalidate
existing indexes, forcing them to be rebuilt (on a RHEL>7 system) if they
contain any values which sort differently under the new collation than the
old one.

More details can be found here, including a query to identify affected
indexes: https://wiki.postgresql.org/wiki/Locale_data_changes

[RHEL] *Version 8 uses the new locale data*. Therefore, caution will be
> necessary when upgrading."

Thanks, Chris.

On Wed, 22 Oct 2025 at 11:53, Bala M <[email-masked]> wrote:

> PostgreSQL 11 setup from *RHEL 7* to *RHEL 9* using *streaming
> replication* (base backup method).
>
> The same indexes work fine on the RHEL 7 (primary) side. However, on the RHEL 9 replica, queries that rely on this index return zero records.
> Rebuilding the indexes  fixed the issue temporarily but we have many indexes  and our DB size is more than 10TB.
>
> *Environment details:*
>
>    -
>
>    Indexes appear and are the same size as per prod  on standby after
>    base backup restore.
>    -
>
> Could this be related to OS-level binary or page layout differences
> between RHEL 7 and RHEL 9 for PostgreSQL 11 binaries?

---

From: Adrian Klaver <[email-masked]>
Date: Wed, 22 Oct 2025 07:31:34 -0700

> replication* (base backup method).

Postgres 11 is two years past EOL per:

https://www.postgresql.org/support/versioning/

-- 
Adrian Klaver
[email-masked]

---

From: Greg Sabino Mullane <[email-masked]>
Date: Wed, 22 Oct 2025 13:35:12 -0400

>    PostgreSQL Version: 11.15
Ouch! Not only is the major version end of life, but that's not even the
latest revision of 11. At this point, you should use logical replication to
migrate from your v11 to a shiny new Postgres v18 on your RHEL9 box. As
Chris pointed out, the indexes are a libc problem, but your bigger problem
is being on v11.15. If you want to trade complexity for time, you could
consider using pg_dump to do the upgrade instead of logical replication.

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

---

From: Ron Johnson <[email-masked]>
Date: Wed, 22 Oct 2025 14:18:38 -0400

On Wed, Oct 22, 2025 at 1:35 PM Greg Sabino Mullane <[email-masked]>
wrote:

> latest revision of 11.

Do pre-built PG11 binaries even exist for RHEL9?

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

---

From: Adrian Klaver <[email-masked]>
Date: Wed, 22 Oct 2025 11:26:47 -0700

On 10/22/25 11:18, Ron Johnson wrote:
> On Wed, Oct 22, 2025 at 1:35 PM Greg Sabino Mullane <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     On Wed, Oct 22, 2025 at 6:53 AM Bala M <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>           *
> 
>     Ouch! Not only is the major version end of life, but that's not even
>     the latest revision of 11.
> 

I believe you can get them here:

https://yum.postgresql.org/repopackages/#archivedrepos

I don't use RH do I can't verify.

---

From: Greg Sabino Mullane <[email-masked]>
Date: Thu, 23 Oct 2025 11:20:38 -0400

>
>
>    -
>
>    *Acceptable downtime:* ~1 day
>    -
>
>    *Logical replication:* Not feasible due to the number of schemas,
>    tables, and overall data volume
>
> I'm not sure why this is not feasible. Can you expand on this?

* For a *15 TB database* with roughly *1 day downtime*, what would be the
> most reliable approach to migrate from *RHEL 7 → RHEL 9* while avoiding
> collation/index corruption issues?

pg_dump is the most reliable, and the slowest. Keep in mind that only the
actual data needs to move over (not the indexes, which get rebuilt after
the data is loaded). You could also mix-n-match pg_logical and pg_dump if
you have a few tables that are super large. Whether either approach fits in
your 24 hour window is hard to say without you running some tests.

* Would using *pg_upgrade* (with --check and --clone options) be safe when
> moving between OS versions with different glibc libraries?

No, you cannot use pg_upgrade for this. It can move your system across
Postgres versions, but across servers/operating systems.

* If we temporarily remain on PostgreSQL 11, is it *mandatory to rebuild
> all indexes* after restoring the base backup on RHEL 9 to ensure data
> consistency? Would running REINDEX DATABASE across all databases be
> sufficient?

Yes, and yes.

* Are there any *community-tested procedures or best practices* for
> migrating large (15 TB+) environments between RHEL 7 and RHEL 9 with
> minimal downtime?

Yes - logical replication is both battle-tested and best practice for such
an upgrade. But with such a large downtime window, investigate pg_dump to
v18. You can find a large table and dump just that one table to start
getting some measurements, e.g. run from the new server:

pg_dump -h my_rhel7_server -d mydb -t mybigtable | psql -h localhost -d
mydb -f -

Make sure log_min_duration_statement is set on the new server to help you
see how long each step takes.

Cheers,
Greg

---

From: Adrian Klaver <[email-masked]>
Date: Thu, 23 Oct 2025 08:49:34 -0700

On 10/23/25 07:51, Bala M wrote:
> Hi All,
> 
> Thank you for the valuable responses regarding the *index corruption 
> issue* we observed during our test migration of a *PostgreSQL 11* 
> environment from *RHEL 7 to RHEL 9* using *streaming replication*.
> 
> Based on the replies, I understand that.

>     Suggested upgrading to a newer PostgreSQL version (e.g., v16 or v18)
>     and using *logical replication* or *pg_dump/restore* for a clean
>     rebuild.

I am not sure version 18 would a good choice at this time, it has just 
been released and has no bug fix releases against it yet. Given the 
other complications in your upgrade, OS upgrade and multi-version jump, 
I don't think you want to add a brand new version to the mix. Postgres 
17 on the other hand has been out a year and has had 6 bug/security 
fixes. It is probably a more stable target version.

---

From: Greg Sabino Mullane <[email-masked]>
Date: Fri, 24 Oct 2025 08:53:47 -0400

On Thu, Oct 23, 2025 at 11:49 AM Adrian Klaver <[email-masked]>
wrote:

> other complications in your upgrade, OS upgrade and multi-version jump, I
> don't think you want to add a brand new version to the mix. Postgres
> 17 on the other hand has been out a year and has had 6 bug/security fixes.
> It is probably a more stable target version.
>

I hear your concern, but I'm pretty confident in v18. Additionally, they
are on version 11 (eleven!) so obviously major upgrades are a rare thing,
so might as well buy themselves another year. :)

Cheers,
Greg

---

From: Greg Sabino Mullane <[email-masked]>
Date: Fri, 24 Oct 2025 08:58:57 -0400

On Thu, Oct 23, 2025 at 10:51 AM Bala M <[email-masked]> wrote:

> Any advice, recommendations, or shared experiences from others who have
> performed similar migrations would be greatly appreciated.

Some related advice: put some system in place to make sure you are always
running the latest revision in whatever major version you end up. See:

In particular, note this part:

*The community considers performing minor upgrades to be less risky than
continuing to run an old minor version.*

You also want to put something in place to make sure your major version
does not  fall so far behind again. You don't need to upgrade every year,
but certainly target a major upgrade every 2-3 years. As you will discover,
the major upgrade process is going to be much easier than this current
upgrade we are talking about in this thread.

Cheers,
Greg

---

From: Adrian Klaver <[email-masked]>
Date: Fri, 24 Oct 2025 07:54:02 -0700

On 10/24/25 05:53, Greg Sabino Mullane wrote:
> On Thu, Oct 23, 2025 at 11:49 AM Adrian Klaver 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     I am not sure version 18 would a good choice at this time, it has
>     just been released and has no bug fix releases against it yet. Given
>     the
>     other complications in your upgrade, OS upgrade and multi-version
>     jump, I don't think you want to add a brand new version to the mix.
>     Postgres

Which is pretty much the criteria for a GA release, confidence that it 
is ready to face the general public. That is not the same thing as it 
being guaranteed bug free. Only time facing the manipulations of said 
public proves how many bugs there are and how bad they are.

> are on version 11 (eleven!) so obviously major upgrades are a rare 
> thing, so might as well buy themselves another year. :)

Major upgrade being the operative phrase here. The OP and company are 
going to face enough new hoops to jump through in the move from 11, 
adding a new release to that mix is pushing it a little too far in my 
estimation.

> 
> Cheers,
> Greg
> 
> --
> Crunchy Data - https://www.crunchydata.com <https://www.crunchydata.com>

---

From: Ron Johnson <[email-masked]>
Date: Fri, 24 Oct 2025 11:00:35 -0400

> public proves how many bugs there are and how bad they are.

"Never trust a .0 release with important data" is just as true in 2025 as
it was in 1985.

That's a chicken and egg problem, though, isn't it?

---

From: Adrian Klaver <[email-masked]>
Date: Fri, 24 Oct 2025 08:51:17 -0700

On 10/24/25 08:00, Ron Johnson wrote:
> On Fri, Oct 24, 2025 at 10:54 AM Adrian Klaver 

> "Never trust a .0 release with important data" is just as true in 2025 
> as it was in 1985.
> 

There is nothing stopping you from setting up a test instance and 
kicking the tires on a new release to see if your setup will work 
correctly. Though the issue in this case is deciding what part of the 15 
TB you would test against or finding the resources to replicate the 
entire setup.

---

From: David Rowley <[email-masked]>
Date: Sat, 25 Oct 2025 11:37:55 +1300

On Sat, 25 Oct 2025 at 04:51, Adrian Klaver <[email-masked]> wrote:
> correctly.

I'd say it's exactly that attitude that causes people to think .0
should be avoided. Beta versions are meant for test instances. It'd be
good if people encouraged their use more often rather than pushing
people to defer til GA.

David

---

From: "David G. Johnston" <[email-masked]>
Date: Fri, 24 Oct 2025 19:00:05 -0400

> good if people encouraged their use more often rather than pushing

I’d say it’s negativity bias that causes this.  Remembering the times
serious bugs were discovered in .0-.2 releases strongly leans one to avoid
.0-.2 releases.  Counteracting that bias with data seems needed.  Yes, it
would be great if more people tested betas; but there is no way for users
to have any idea of how much or how effective such efforts have been for a
given release.

If someone comes here asking for advice I’m going to be conservative.  The
ones who are going to be pushing the envelope or doing beta testing are not
asking for this kind of advice.  They do it likely to benefit from new
features and some subset does it to be the canary for the community because
they can handle the rare negative outcome.

David J.

---

From: Adrian Klaver <[email-masked]>
Date: Fri, 24 Oct 2025 17:40:03 -0700

> good if people encouraged their use more often rather than pushing

1) From previous posts to this list folks have mentioned their 
organizations prohibit touching anything less then a GA or maybe a late 
RC. That comes from on high and I doubt the folks issuing the orders are 
on this list.

2) The attitude comes from lessons learned in the School of Hard Knocks. 
Until someone or someones can guarantee a new GA release will not eat 
your data or spring security leaks then the prudent thing to do is wait 
to see what happens when it hits the world at large. I learned this 
lesson, pitfalls of jumping into something new, across fields outside of 
software as well. In other words 'new and improved' is not always the 
case, see 737 MAX as case in point.

3) Progress happens and you need to keep up. A little caution is good 
thing though, especially if you are the one who is being held 
responsible for any adverse outcomes.

---

From: David Rowley <[email-masked]>
Date: Sat, 25 Oct 2025 14:06:15 +1300

> RC. That comes from on high and I doubt the folks issuing the orders are

That seems bizarre to me. If they want new releases of PostgreSQL to
be as stable as possible as early as possible, then beta and RC are
much better times to test than .0 is.

> case, see 737 MAX as case in point.

I don't see why this reason is applicable at all to my statement. I
didn't state that everyone should go and run with .0. I said we
shouldn't encourage people to test beta and RC versions, as if they
don't do that then .0 won't be as stable as if they did test (and
report issues). It seems like simple cause and effect to me.

> responsible for any adverse outcomes.

We're talking test servers here. I assume they can be recreated
without too much trouble.

David

---

From: Adrian Klaver <[email-masked]>
Date: Fri, 24 Oct 2025 21:36:56 -0700

> report issues). It seems like simple cause and effect to me.

I am not following, from your previous post:

"Beta versions are meant for test instances. It'd be
good if people encouraged their use more often rather than pushing
people to defer til GA"

That seems to be the opposite of what you say above.

> without too much trouble.

Yes, but the OP was talking about upgrading a production database 
directly to 18. That was what my reply was referring to and what I was 
counseling against.

---

From: David Rowley <[email-masked]>
Date: Sat, 25 Oct 2025 17:50:59 +1300

> That seems to be the opposite of what you say above.

I think you think that because you misunderstood what I said in [1].
I'll rephrase it for you:

Because people promote the .0 as not yet production-ready, it means
that fewer people bother testing with beta and RC versions. Lack of
beta testing is what causes .0 to contain more bugs than it otherwise
might, so my suggestion is that we should be encouraging people to run
beta and RC in their test environments to try to increase the
stability of .0 versions.

I struggle to imagine anyone with any respect for the PostgreSQL
project disagreeing with that, so I suspect you and Johnson must have
misunderstood.

Also, I was highlighting that I agreed with the paradox pointed out by Ron.

Clear?

> directly to 18. That was what my reply was referring to and what I was

You should pay more attention to the quotes above my reply. I don't
see any of my replies quoting anything about upgrading a production
database to 18.0.

David

[1] https://postgr.es/m/CAApHDvo5pLSwD4JVxW8t94Vc88djKrrSyrL-GbQJ=[email-masked]

---

From: Adrian Klaver <[email-masked]>
Date: Sat, 25 Oct 2025 07:21:16 -0700

> stability of .0 versions.

Alright that I understand, though not necessarily agree with. I would 
say lack of testing has more to do with time/money management. 
Organizations don't want to spend either until: 1) They see the dust 
settle on what is going to end up in the release. 2) Whether there is 
anything interesting enough to invest both in moving to a new release. 
Maybe there is a compelling argument that can be made to get those 
organizations off the fence. I just don't what it is as you would have 
to convince them to spend time and money rather then just wait and let 
the community as a whole do the work.

> project disagreeing with that, so I suspect you and Johnson must have

No we didn't misunderstand, we where responding to what the OP was 
proposing which was jumping a production instance from 11 --> 18. That 
is a different case then promoting testing of in beta's and rc's.

---

From: Ron Johnson <[email-masked]>
Date: Sat, 25 Oct 2025 10:39:35 -0400

On Sat, Oct 25, 2025 at 10:21 AM Adrian Klaver <[email-masked]>
wrote:

> the community as a whole do the work.\
>

Contractual requirements to not run EOL software are a strong motivator to
migrate to newer versions of OS and RDBMS.