Date: Mon, 22 Apr 2024 11:50:27 +0000
To: "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
From: Nicolas Seinlet <nicolas@seinlet.com>
Subject: Re: Failing streaming replication on PostgreSQL 14
Message-ID: <5DyhufZPA9iljX8zsyrAw8zCw3wg4GsrKTodhOtvS-tJOmYHKoIiHEdKH4DCbkTA14fDzriyIJa0sFJgVN8W_HFDEIQY6nNwujdMRweJmzI=@seinlet.com>
In-Reply-To: <iCjVHvcOYXIzd9YMUrvJkbJoswyKKAs8aQROIv41Oa-2Aj1H4YdvX_WWd_0-5czna8Q44XFZKbW4FFJ7x4dEcgwQ-_egpXN9STRTE1Hd8Yc=@seinlet.com>
References: <202404151649.ypmcghdjuecs@alvherre.pgsql> <iCjVHvcOYXIzd9YMUrvJkbJoswyKKAs8aQROIv41Oa-2Aj1H4YdvX_WWd_0-5czna8Q44XFZKbW4FFJ7x4dEcgwQ-_egpXN9STRTE1Hd8Yc=@seinlet.com>
Feedback-ID: 32582315:user:proton
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha512; boundary="------e381136652bd7f306bb2a8d110610bbf7e403f8df8efa4d27f78612fd9ccfe82"; charset=utf-8
Archived-At: <https://www.postgresql.org/message-id/5DyhufZPA9iljX8zsyrAw8zCw3wg4GsrKTodhOtvS-tJOmYHKoIiHEdKH4DCbkTA14fDzriyIJa0sFJgVN8W_HFDEIQY6nNwujdMRweJmzI%3D%40seinlet.com>
Precedence: bulk

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--------e381136652bd7f306bb2a8d110610bbf7e403f8df8efa4d27f78612fd9ccfe82
Content-Type: multipart/mixed;boundary=---------------------08177b1157db7f26b6cdbe0d2d3e68aa

-----------------------08177b1157db7f26b6cdbe0d2d3e68aa
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;charset=utf-8

Hi,

facing the same situation again, but this time, analyzing the wal with xxd=
 shows a different pattern. I had no blocks of 0000.

The output of pg_waldump is:
pg_waldump: fatal: error in WAL record at 11C/93F9FF70: invalid magic numb=
er 0000 in log segment 000000010000011C00000093, offset 16384000

The output of xxd -C16 is

00f9ff60: b364 0079 6e61 6d69 6320 6c80 0300 0000  .d.ynamic l.....
00f9ff70: 4000 0000 6659 a406 60f7 f993 1c01 0000  @...fY..`.......
00f9ff80: 000b 0000 82b3 8d9b 0020 1000 7f06 0000  ......... ......

I'm still unable to determine the cause of the issue, nor if the issue is =
on the primary server sending a corrupted wal segment, or on the secondary=
 receiving a corrupted wal segment, or the openzfs filesystem on the prima=
ry allowing wal_sender to read still-not-written wal segment, or ...

Is there any log option I can add on the two clusters to help me locate th=
e issue's origin?

thanks,

Nicolas.

On Tuesday, April 16th, 2024 at 09:56, Nicolas Seinlet <nicolas@seinlet.co=
m> wrote:

> =


> =


> Hello,
> =


> > What exactly is "cyphered ZFS"? Can you reproduce the problem with som=
e
> > other filesystem? If it's something very unusual, it might well be a
> > bug in the filesystem.
> =


> =


> The filesystem is openzfs with native aes-256-gcm encryption:
> https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#encr=
yption
> =


> I've not tested if we get the same issue on another filesystem.
> =


> I don't face the issue on Ubuntu 20.04/openzfs 0.8/PostgreSQL 12, but I =
have fewer systems with this deployment.
> On Ubuntu 22.04/openzfs 2.1.5/PostgreSQL 14, I face the issue from time =
to time, without knowing what triggers the error.
> =


> thanks for helping,
> =


> Nicolas.


-----------------------08177b1157db7f26b6cdbe0d2d3e68aa--

--------e381136652bd7f306bb2a8d110610bbf7e403f8df8efa4d27f78612fd9ccfe82
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: ProtonMail

wnUEARYKACcFgmYmTvMJkJG8s0TkXG4EFiEEyut/r/ADqlpVUS/JkbyzRORc
bgQAAIzHAP0dKSHqgWu5T5utxH8Bje5PqvkB49jo0mG5DJaFMSwX8wEA08nA
QNeU6Q0jDbpkMxMhZnSQAYZ0s96ytIu2Z1UTEAg=
=jObv
-----END PGP SIGNATURE-----


--------e381136652bd7f306bb2a8d110610bbf7e403f8df8efa4d27f78612fd9ccfe82--