Date: Mon, 18 Apr 2022 10:11:05 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
	SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>,
	Euler Taveira <euler@eulerto.com>
Subject: Re: pg_walcleaner - new tool to detect, archive and delete the
 unneeded wal files (was Re: pg_archivecleanup - add the ability to detect,
 archive and delete the unneeded wal files on the primary)
Message-ID: <20220418141105.GK10577@tamriel.snowman.net>
References: <CALj2ACWrryWEbzn3gU9yqyarY5ptsUd8wrLwNgfv+Xn4tepRJQ@mail.gmail.com>
 <66f58359-6241-47fd-bf25-4d105a00523c@www.fastmail.com>
 <20211229135710.GH15820@tamriel.snowman.net>
 <CALj2ACXfx48SvnPRS_P3OrXsjxgCXXE4u4ROF3pq3r-mkdz61g@mail.gmail.com>
 <CALj2ACV9KTXDO=Vjf4K6v-b58x3XcyWDDzN5su2EK0gVWEPAug@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="2dziexqPTSneikDd"
Content-Disposition: inline
In-Reply-To: <CALj2ACV9KTXDO=Vjf4K6v-b58x3XcyWDDzN5su2EK0gVWEPAug@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: <https://www.postgresql.org/message-id/20220418141105.GK10577%40tamriel.snowman.net>
Precedence: bulk


--2dziexqPTSneikDd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Bharath Rupireddy (bharath.rupireddyforpostgres@gmail.com) wrote:
> Thanks for the comments. Here's a new tool called pg_walcleaner which
> basically deletes (optionally archiving before deletion) the unneeded
> WAL files.
>=20
> Please provide your thoughts and review the patches.

Alright, I spent some more time thinking about this and contemplating
what the next steps are... and I feel like the next step is basically
"add a HINT when the server can't start due to being out of disk space
that one should consider running pg_walcleaner" and at that point... why
aren't we just, uh, doing that?  This is all still quite hand-wavy, but
it sure would be nice to be able to avoid downtime due to a broken
archiving setup.  pgbackrest has a way of doing this and while we, of
course, discourage the use of that option, as it means throwing away
WAL, it's an option that users have.  PG could have a similar option.
Basically, to archive_command/library what max_slot_wal_keep_size is for
slots.

That isn't to say that we shouldn't also have a tool like this, but it
generally feels like we're taking a reactive approach here rather than a
proactive one to addressing the root issue.

Thanks,

Stephen

--2dziexqPTSneikDd
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJiXXF4AAoJEO1sijiDR2RVsaYQAKqU9fOApyFdPlPyg8rEkv7y
X5JVJAfhTHtiGbTmKkyKIfK8k6uW5qfZ7CwFSsJ6u8iJWKj9rK/qBTNhwxNrBmWd
czzObcAogfV9LVF5Nt62ShgXWFeX5yaLh9WnqywnohYfah72HzzQNcG5sPNkAg5j
TJdo9rSMzexUSk4MLDKX9nS4BuSKS14CEK3lrraiy6nLUnnmKIWfEvmJ5OdQ05xk
Z6EzuNvsA875N3R8EvPFQ3b1cYBtRBlCQEmN3wplyIrVGDsRsK4zflTSTYXFeceN
gxgyE07873G1N/mAz9uJj0qtYF1tOMGErEzIr3BcHIhxkJ/hTX0LJkZAt4yYUUXr
rYpX7n1xJb00vqQwV9vK2jE19fLI32tM00vLkmRuhBL3bzMykbwsVSsFF+QTuIRL
Ilct+h7K1CzaLZjYburXfy8jUerYpOZMt7zg+Et2ZmAJGE6zYA4kr67rph6BxWW6
io/f88EIVtLngQB+c2uCp5b1WFppmLJbazeFtkZ79P/zT2HELhQKj1mvybtVkE+I
rYKuv/BcrSqlAPDtKlYsGb9oFYkL0BDG7lB9m5qFh/lWuT3IIi1jvl+XEB96q8sL
dfTxGQgsOJiwLHGKh7qal9ib7E2iJByrcWY2z6pZZvWK0gxv63/Sp5qMYDiSVISG
F1gwEzK2iMfAtiIpdWGK
=S15/
-----END PGP SIGNATURE-----

--2dziexqPTSneikDd--