Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (1.0)
Subject: Re: Logical replication and wal segment retention
From: John Scalia <jayknowsunix@gmail.com>
In-Reply-To: <59567e0a-78e4-ed1c-f5b1-033aafcfb68a@matrix.gatewaynet.com>
Date: Wed, 27 Feb 2019 08:29:09 -0500
Cc: pgsql-admin@lists.postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <75D08FE7-36EC-4721-9382-DF8AF508644D@gmail.com>
References: <695C083D-AFB5-431B-B022-ED442CE94794@gmail.com>
 <20190227125242.6ixy4jkzcuj55cdl@vault.lan>
 <59567e0a-78e4-ed1c-f5b1-033aafcfb68a@matrix.gatewaynet.com>
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Precedence: bulk

I thought as much. The basic problem, however, is that I never created the l=
ogical slots. The IIDR application did that all by itself, and after it term=
inated, it did not bother to remove the slots. So, my disk monitor threw up w=
hen the WAL file system began to fill up. I was trying then to figure out wh=
y it did that.

Sent from my iPhone

> On Feb 27, 2019, at 7:57 AM, Achilleas Mantzios <achill@matrix.gatewaynet.=
com> wrote:
>=20
>> On 27/2/19 2:52 =CE=BC.=CE=BC., Johannes Truschnigg wrote:
>> Hi Jay,
>>=20
>>> On Wed, Feb 27, 2019 at 07:40:26AM -0500, John Scalia wrote:
>>> Hello, folks,
>>>=20
>>> Yesterday, I had a small file system fill up, due to some logical
>>> replication testing we had been performing. We had been testing IBM=E2=80=
=99s IIDR
>>> system and apparently it had built a logical replication slot on my serv=
er.
>>> When the test was completed, nobody removed the slot, so WAL segments
>>> stopped being dropped. Now I can understand the difficulty separating wh=
at
>>> physical versus logical replication needs from the WAL segments, but as
>>> logical replication is database specific not cluster wide, this behavior=
 was
>>> a little unexpected, since the WAL segments are cluster wide. Are WAL
>>> segments going to pile up whenever something drops a logical replication=

>>> connection? I=E2=80=99ve seen it, but it seems like this could be a bad t=
hing.
>> Since Logical Replication is piggybacked on Physical Replication, you can=
not
>> use the first without having the latter. And yes, what you experienced is=
 one
>> of the dangers of using replication slots when having a busy database (i.=
e.
>> producing lots of WAL) and a filesystem with little excess space. Under t=
hese
>> circumstances, it is imperative to monitor for (and alert on) anything go=
ing
>> awry with your replication slot consumers, and/or the size of your wal/xl=
og
>> directory. It's a feature of replication slots to work that way - but one=
 that
>> may end up biting you.
> A logical approach for replication slots would be to accept a parameter re=
garding max WAL files to retain, after which newer WALs will be removed and t=
he primary server saved.  Pretty much like : --archive-push-queue-max argume=
nt of pgbackrest .
>>=20
>=20
>=20
> --=20
> Achilleas Mantzios
> IT DEV Lead
> IT DEPT
> Dynacom Tankers Mgmt
>=20
>=20