Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.400.21\))
Subject: Re: Streaming replication and WAL archive interactions
From: Jaroslav Novikov <njrslv@yandex-team.ru>
In-Reply-To: <D4B53AE3-B7AF-4BE6-9CB6-44956B05DE72@yandex-team.ru>
Date: Tue, 3 Mar 2026 15:06:58 +0300
Cc: hlinnaka@iki.fi,
 Michael Paquier <michael.paquier@gmail.com>,
 Robert Haas <robertmhaas@gmail.com>,
 Venkata Balaji N <nag1010@gmail.com>,
 Andres Freund <andres@anarazel.de>,
 Fujii Masao <masao.fujii@gmail.com>,
 Borodin Vladimir <root@simply.name>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>,
 nkak@vmware.com,
 Roman Khapov <rkhapov@yandex-team.ru>,
 Kirill Reshke <reshkekirill@gmail.com>,
 ShirishaRao@vmware.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <9A271666-C8DA-455E-B5C7-48FF01CC72AB@yandex-team.ru>
References: <548AF1CB.80702@vmware.com>
 <689EB259-44C2-4820-B901-4F6B1C55A1E4@simply.name>
 <549083D6.1000301@vmware.com>
 <CAHGQGwGJzp-QS7BODiv1uc291gAKtjzzCPb_nzUTxYKJhLsUCA@mail.gmail.com>
 <54949108.3030109@vmware.com>
 <CAEyp7J9Hy8Q__FbGeR5skjk7d0dvLC+KLXB3JUuWrXXdJ5O+Wg@mail.gmail.com>
 <552FA38F.9060005@iki.fi>
 <CAB7nPqQE179yogtg+nKvdwt9KROxTyt-EjumKOMbuXQtea5r3w@mail.gmail.com>
 <5535FE71.1010905@iki.fi>
 <CAB7nPqS=V=LF-JZsv9eGF84joQ3jxugt4gkOK6kD1CiB98_vWg@mail.gmail.com>
 <55362CAD.2000207@iki.fi>
 <CA+TgmoZhcbyDBr8UdGu=y5kmnVcSwcnaBqqV=UnP+oCGiNQGeQ@mail.gmail.com>
 <CAB7nPqRhsOHSc3zyL0qw4-CRphb-9Nm4sXhRCA4iKHUhkdFysA@mail.gmail.com>
 <553741FE.1080403@iki.fi>
 <CAB7nPqSak3K65B+Pzp_qGrd0rO9L=1T5FLUkPEXRqtgvV9zXEw@mail.gmail.com>
 <554CB84E.3070406@iki.fi> <5550D20D.6090703@iki.fi>
 <D4B53AE3-B7AF-4BE6-9CB6-44956B05DE72@yandex-team.ru>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Archived-At: 
 <https://www.postgresql.org/message-id/9A271666-C8DA-455E-B5C7-48FF01CC72AB%40yandex-team.ru>
Precedence: bulk


> On 12 Feb 2026, at 09:56, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>=20
> Hi Heikki,
>=20
> There=E2=80=99s a nearby thread [0] (about 10 years later) where I=E2=80=
=99m working on a problem your patch from this thread helps solve.
>=20
> In datacenter large outages, 1=E2=80=932% of clusters end up with gaps =
in their PITR timeline.
> In HA setups, when the primary is lost, some WAL can be missing from =
the archive even though it was streamed to the standby. Many HA tools =
(PGConsul, Patroni, etc.) try to re-archive from the standby, but those =
WAL files may already have been removed.
>=20
> Your =E2=80=9Cshared=E2=80=9D archive mode addresses this: the standby =
keeps WAL until it=E2=80=99s archived. archive_mode=3Dalways plus an =
archive tool can work, but it=E2=80=99s expensive. In WAL-G, for =
example, the archive command does a GET on the standby=E2=80=99s WAL, =
then decrypts and compares. Switching to HEAD would reduce cost in some =
clouds but still adds cost.
>=20
> Another option is coordinating archiving outside Postgres, but that =
would mean building distributed coordination into the archive tool.
>=20
> Shared archive mode tackles this in Postgres itself.
>=20
> I=E2=80=99ve retrofitted your patch, incorporated ideas from the =
Greenplum work [1], and made some improvements.
>=20
> The patchset has three parts:
> * Rebase + tests =E2=80=93 Your original patch, rebased, with tests =
added.
> * Timeline switching =E2=80=93 Correct handling of timeline switches =
in archive status updates.
> * Avoid directory scans =E2=80=93 Skip scanning archive_status when =
possible, which was costly in WAL-G setups.
>=20
> What do you think?
>=20
> Best regards, Andrey Borodin.
>=20
> =
<v4-0001-Add-archive_mode-shared-for-coordinated-WAL-archi.patch><v4-0003-=
Optimize-ProcessArchivalReport-to-avoid-directory.patch><v4-0002-Mark-ance=
stor-timeline-WAL-segments-as-archived.patch>

Hi Andrey,

Adding the missing references [0] and [1].

[0] https://www.postgresql.org/message-id/5550D20D.6090703%40iki.fi
[1] =
https://github.com/open-gpdb/gpdb/commit/4f2db1929df1b5eed28f3350595563609=
6bb4e8b

Best, Jaroslav Novikov.