pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedFrom: Johannes Truschnigg <johannes@truschnigg.info>
To: Mohan NBSPS <mohan.nbs.ont@gmail.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Postgresql 9.5: Streaming Replication: Secondaries Fail To Start Post WAL Error
Date: Tue, 28 May 2024 21:11:36 +0200
Message-ID: <ZlYsaFDpLdR9SgAM@vault.lan> (raw)
In-Reply-To: <CAPCvfWdaymCYkLz-AseQLwGj8qh6pT1Kx73LoS-wkNNrukEX_Q@mail.gmail.com>
References: <CAPCvfWcm0JDC+q54MSW7N90PYvh+PefaP6SxfonbkGcUwpS1+g@mail.gmail.com>
<ZlYmsG4NX7WjzFs-@vault.lan>
<CAPCvfWdaymCYkLz-AseQLwGj8qh6pT1Kx73LoS-wkNNrukEX_Q@mail.gmail.com>
On Tue, May 28, 2024 at 03:00:23PM -0400, Mohan NBSPS wrote:
> [...]
> Thank you Johannes for the advice.
>
> We are looking at moving to 16.
> We did not implement slots to avoid disk space issues on primary (possible
> network disconnect may fill up primary `pg_xlog`).
Yes, replication slots can interrupt your primary. Relying on
wal_keep_segments alone can kill your replicas. Having a WAL archive can be
the best of both worlds, but also needs careful monitoring and management.
> We have changed the WAL settings to retain more WAL files.
>
> Since we have not seen this issue before, (have been running postgresql for
> over 10 years), what kind
> of scenario would trigger this ?
Every time you interrupt the replication stream (such as when a replica
reboots, or its postgres master process is stopped), you enter a race
condition between WAL segments accumulating on the primary, and the
replication stream to pick up again once the replica is up once more. So if,
during your replica restart, enough WAL was produced to exceed
wal_keep_segments, the lineage is broken, and the replica cannot ever catch up
again.
Also, the "invalid resource manager" log line you reported *might* hint at
data corruption in your WAL segments. I think that data checksums and WAL
compression could both make detection of such conditions more reliable.
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
Attachments:
[application/pgp-signature] signature.asc (832B, ../ZlYsaFDpLdR9SgAM@vault.lan/2-signature.asc)
download
view thread (8+ messages) latest in thread
Message-ID: <ZlYsaFDpLdR9SgAM@vault.lan>
Permalink: ../ZlYsaFDpLdR9SgAM@vault.lan/
Also on: postgresql.org/message-id/ZlYsaFDpLdR9SgAM@vault.lan
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: johannes@truschnigg.info, mohan.nbs.ont@gmail.com, pgsql-admin@lists.postgresql.org
Subject: Re: Postgresql 9.5: Streaming Replication: Secondaries Fail To Start Post WAL Error
In-Reply-To: <ZlYsaFDpLdR9SgAM@vault.lan>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox