Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sFLTA-001Pw1-GM for pgsql-general@arkaria.postgresql.org; Thu, 06 Jun 2024 22:20:29 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sFLSA-004fVL-CT for pgsql-general@arkaria.postgresql.org; Thu, 06 Jun 2024 22:19:27 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sFLS9-004fUq-QO for pgsql-general@lists.postgresql.org; Thu, 06 Jun 2024 22:19:26 +0000 Received: from mail-ej1-x631.google.com ([2a00:1450:4864:20::631]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sFLS7-0003tL-KD for pgsql-general@lists.postgresql.org; Thu, 06 Jun 2024 22:19:25 +0000 Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a69607c6ccaso166426966b.2 for ; Thu, 06 Jun 2024 15:19:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717712361; x=1718317161; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=xqXck7ja8aa+V5yL83cdFAHFQZSqW3vntfWO68/rqXQ=; b=CEiHwHLdIqmbOp6GlliCvAlqu8zfO6ivmD+1Jobif0fmE+QymZfMixRRvPwuQ86j9j BiUS9Y8pcZTcK58oX3dT66lTQR7Wt9lImUpXXxg4PgptyEOPIrQK1v7jEjRjM5f3cmXB Oocjb6l5dF58ximgMWMAtMNILbl4oii9/ZBQj3bxv18XD1Hjqg6ZHd9w+sZ6ytuFYCq4 5fVPODXIGj/cjuRASSPxO0nTSPqMQOYQCeowHs3zaoGcG82Np5PkxhI4K78yLJ1PS61N fircV2oOGiD/qD3AbENLshz3hxVKB1St3JekHHlcmWQ/JOrxezK2FtlgBF6ddITY9K8K IHmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717712361; x=1718317161; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xqXck7ja8aa+V5yL83cdFAHFQZSqW3vntfWO68/rqXQ=; b=MgzRnPejv4MEx8jmkmm0J31RffeLMO6+3bokKVdb+j6jGRc4/enEKX+1tJWt1r0IYm mhbRS4XyngMtTHG/1w8FihegpLc9Adbv+87XVOt9ac+N0gWa560os/qU3EE90/2fyRJY u8qnXqramaP11ocYyFnJ58g7Jg4kMU6nra5UXsna+Ngq5knezRDsqnZd/sd/kHHBL2yO /uWgv2ATs7X5C484qNzGqtQX6QXl7/erUipGpoAIPYp32bV2rwmaHv8Lmy4iH8XldQwf HuNJf/GVh9qFaJgYUnEd2lOqDHgoomvEY7eMSGiG6o452A/Cz1n2hyGBZ9a3UIsHiaij Uqgw== X-Gm-Message-State: AOJu0Yy8zPagpvSpEO4SuyCHVFJkxaoqV/tW5GXVguf9VUYNFihCE0ZX 0jvNr/8VVD0PnjC8MBBQDA8niW4rijnvr+XiWIUwk/wZQNSh72FAzEQKdxgMb+5ZaUxofQBdMST DW4eOYUHAmnPHtBJoixsHGVsjUj/AiE4b X-Google-Smtp-Source: AGHT+IF9S/3hrWqiBdXp+KD1Z3immi5ZkHdkK+SnLlMiGh6ycPUuCvLvrrVNxn+KgzJDnAw05wMG4BAHgP5UKW20iAs= X-Received: by 2002:a17:906:718:b0:a68:b8b9:b924 with SMTP id a640c23a62f3a-a6cd7a843a1mr53078566b.38.1717712360467; Thu, 06 Jun 2024 15:19:20 -0700 (PDT) MIME-Version: 1.0 References: <1628637f-419f-4f6a-9cb6-07af90cd0bc4@aklaver.com> <1c0273f5-a90a-48f6-b51f-fe15c16fa1c6@aklaver.com> In-Reply-To: <1c0273f5-a90a-48f6-b51f-fe15c16fa1c6@aklaver.com> From: Koen De Groote Date: Fri, 7 Jun 2024 00:19:08 +0200 Message-ID: Subject: Re: Questions on logical replication To: Adrian Klaver Cc: PostgreSQL General Content-Type: multipart/alternative; boundary="00000000000012789b061a401395" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --00000000000012789b061a401395 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'll give them a read, though it might take a few weekends Meanwhile, this seems to be what I'm looking for: From https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICA= TION-SLOTS " Replication slots provide an automated way to ensure that the primary does not remove WAL segments until they have been received by all standbys, and that the primary does not remove rows which could cause a recovery conflict even when the standby is disconnected." I'm reading that as: "if there is a replication slot, if the standby is disconnected, WAL is kept" And if we know WAL is kept in the "pg_wal" directory, that sounds like it could slowly but surely fill up disk space. But again, I'll give them a read. I've read all of logical replication already, and I feel like I didn't get my answer there. Thanks for the help Regards, Koen De Groote On Thu, Jun 6, 2024 at 12:19=E2=80=AFAM Adrian Klaver wrote: > On 6/5/24 14:54, Koen De Groote wrote: > > https://www.postgresql.org/docs/current/wal-configuration.html > > > > > > "Checkpoints are points in the sequence of transactions at which it > is > > guaranteed that the heap and index data files have been updated wit= h > > all > > information written before that checkpoint. At checkpoint time, all > > dirty data pages are flushed to disk and a special checkpoint recor= d > is > > written to the WAL file. (The change records were previously flushe= d > to > > the WAL files.) In the event of a crash, the crash recovery procedu= re > > looks at the latest checkpoint record to determine the point in the > WAL > > (known as the redo record) from which it should start the REDO > > operation. Any changes made to data files before that point are > > guaranteed to be already on disk. Hence, after a checkpoint, WAL > > segments preceding the one containing the redo record are no longer > > needed and can be recycled or removed. (When WAL archiving is being > > done, the WAL segments must be archived before being recycled or > > removed.)" > > > > > > And this is the same for logical replication and physical replication, = I > > take it. > > High level explanation, both physical and logical replication use the > WAL files as the starting point. When the recycling is done is dependent > on various factors. My suggestion would be to read through the below to > get a better idea of what is going. There is a lot to cover, but if you > really want to understand it you will need to go through it. > > Physical replication > > https://www.postgresql.org/docs/current/high-availability.html > > 27.2.5. Streaming Replication > 27.2.6. Replication Slots > > Logical replication > > https://www.postgresql.org/docs/current/logical-replication.html > > WAL > > https://www.postgresql.org/docs/current/wal.html > > > > > > > Thus, if a leader has a standby of the same version, and meanwhile > > logical replication is being done to a newer version, both those > > replications are taken into account, is that correct? > > Yes, see links above. > > > > And if it cannot sync them, due to connectivity loss for instance, the > > WAL records will not be removed, then? > > Depends on the type of replication being done. It is possible for > physical replication to have WAL records removed that are still needed > downstream. > > From > > > https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLI= CATION > > "If you use streaming replication without file-based continuous > archiving, the server might recycle old WAL segments before the standby > has received them. If this occurs, the standby will need to be > reinitialized from a new base backup. You can avoid this by setting > wal_keep_size to a value large enough to ensure that WAL segments are > not recycled too early, or by configuring a replication slot for the > standby. If you set up a WAL archive that's accessible from the standby, > these solutions are not required, since the standby can always use the > archive to catch up provided it retains enough segments." > > This is why it is good idea to go through the links I posted above. > > > > > Regards, > > Koen De Groote > > > > > -- > Adrian Klaver > adrian.klaver@aklaver.com > > --00000000000012789b061a401395 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I'll give them a read, though it might take a few= weekends

Meanwhile, this seems to be what I'm= looking for:


" Replication slots provide an automated way to ensure that the primary=20 does not remove WAL segments until they have been received by all=20 standbys, and that the primary does not remove rows which could cause a recovery conflict even when the standby is disconnected."

I'm reading that as: "if there is a replica= tion slot, if the standby is disconnected, WAL is kept"

=
And if we know WAL is kept in the "pg_wal" directory, = that sounds like it could slowly but surely fill up disk space.
<= br>

But again, I'll give them a read. I've= read all of logical replication already, and I feel like I didn't get = my answer there.

Thanks for the help


Regards,
Koen De Groote
<= /div>
O= n Thu, Jun 6, 2024 at 12:19=E2=80=AFAM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 6/5/24 14:54, Koen D= e Groote wrote:
>=C2=A0 =C2=A0 =C2=A0https://www.po= stgresql.org/docs/current/wal-configuration.html
>=C2=A0 =C2=A0 =C2=A0<https://ww= w.postgresql.org/docs/current/wal-configuration.html>
>
>=C2=A0 =C2=A0 =C2=A0"Checkpoints are points in the sequence of tra= nsactions at which it is
>=C2=A0 =C2=A0 =C2=A0guaranteed that the heap and index data files have = been updated with
>=C2=A0 =C2=A0 =C2=A0all
>=C2=A0 =C2=A0 =C2=A0information written before that checkpoint. At chec= kpoint time, all
>=C2=A0 =C2=A0 =C2=A0dirty data pages are flushed to disk and a special = checkpoint record is
>=C2=A0 =C2=A0 =C2=A0written to the WAL file. (The change records were p= reviously flushed to
>=C2=A0 =C2=A0 =C2=A0the WAL files.) In the event of a crash, the crash = recovery procedure
>=C2=A0 =C2=A0 =C2=A0looks at the latest checkpoint record to determine = the point in the WAL
>=C2=A0 =C2=A0 =C2=A0(known as the redo record) from which it should sta= rt the REDO
>=C2=A0 =C2=A0 =C2=A0operation. Any changes made to data files before th= at point are
>=C2=A0 =C2=A0 =C2=A0guaranteed to be already on disk. Hence, after a ch= eckpoint, WAL
>=C2=A0 =C2=A0 =C2=A0segments preceding the one containing the redo reco= rd are no longer
>=C2=A0 =C2=A0 =C2=A0needed and can be recycled or removed. (When WAL ar= chiving is being
>=C2=A0 =C2=A0 =C2=A0done, the WAL segments must be archived before bein= g recycled or
>=C2=A0 =C2=A0 =C2=A0removed.)"
>
>
> And this is the same for logical replication and physical replication,= I
> take it.

High level explanation, both physical and logical replication use the
WAL files as the starting point. When the recycling is done is dependent on various factors. My suggestion would be to read through the below to get a better idea of what is going. There is a lot to cover, but if you really want to understand it you will need to go through it.

Physical replication

https://www.postgresql.org/docs/curren= t/high-availability.html

27.2.5. Streaming Replication
27.2.6. Replication Slots

Logical replication

https://www.postgresql.org/docs/curr= ent/logical-replication.html

WAL

https://www.postgresql.org/docs/current/wal.html=



>
> Thus, if a leader has a standby of the same version, and meanwhile > logical replication is being done to a newer version, both those
> replications are taken into account, is that correct?

Yes, see links above.


> And if it cannot sync them, due to connectivity loss for instance, the=
> WAL records will not be removed, then?

Depends on the type of replication being done. It is possible for
physical replication to have WAL records removed that are still needed
downstream.

From

https://www.postgresq= l.org/docs/current/warm-standby.html#STREAMING-REPLICATION

"If you use streaming replication without file-based continuous
archiving, the server might recycle old WAL segments before the standby has received them. If this occurs, the standby will need to be
reinitialized from a new base backup. You can avoid this by setting
wal_keep_size to a value large enough to ensure that WAL segments are
not recycled too early, or by configuring a replication slot for the
standby. If you set up a WAL archive that's accessible from the standby= ,
these solutions are not required, since the standby can always use the
archive to catch up provided it retains enough segments."

This is why it is good idea to go through the links I posted above.

>
> Regards,
> Koen De Groote
>


--
Adrian Klaver
adrian.klave= r@aklaver.com

--00000000000012789b061a401395--