Date: Mon, 8 Feb 2021 16:28:40 -0500
From: Stephen Frost <sfrost@snowman.net>
To: Victor Sudakov <vas@sibptus.ru>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: wal-g (https://github.com/wal-g/wal-g) reliability
Message-ID: <20210208212840.GW27507@tamriel.snowman.net>
References: <YB0VuB8RJAKvTI6+@admin.sibptus.ru>
 <CANNMO++t7hLG2Uk-M61ohXGNhBYszSPL8vn=zP09tcRv15QB4A@mail.gmail.com>
 <YCDKyYzrv4HQZz9t@admin.sibptus.ru>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="3zw9Zue9ODpaeBn+"
Content-Disposition: inline
In-Reply-To: <YCDKyYzrv4HQZz9t@admin.sibptus.ru>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--3zw9Zue9ODpaeBn+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Victor Sudakov (vas@sibptus.ru) wrote:
> Nikolay Samokhvalov wrote:
> > In short, it's reliable and battle-tested. It's used in  companies such=
 as
> > Yandex.Cloud and GitLab.com, successfully.
>=20
> Nikolay, don't you mind another question?=20
>=20
> I'm used to the fact that "pg_basebackup -X" creates a self-sufficient
> backup of a cluster which can be started right away as it contains all
> the WAL files required for recovery. `touch recovery.signal` is never nec=
essary,
> and `touch standby.signal` is optional (when you do PITR etc).
>=20
> It's not the case with wal-g, the result of the `wal-g backup-fetch`=20
> command requires `touch recovery.signal` and a restore_command
> configured to fetch WALs from the wal-g storage.
>=20
> I have also noticed that wal-g keeps pg_control in a separate tar
> archive, and keeps a lot of metadata.
>=20
> The questions are:=20
>=20
> 1. If the metadata in the wal-g storage ever becomes corrupt, will I be
> able to restore the database manually from the archives in
> $WALG_*_PREFIX/{basebackups,wal}_005/ ?
>=20
> 2. Is there a `wal-g backup-fetch` option for truly self-sufficient
> restoration?

Interesting that you ask this!  I say that because it's actually a case
that pgbackrest contemplated and explicitly added support for-
specifically, if you set --archive-copy (or archive-copy=3Dtrue) and
disable compression (or decompress everything before you start PG), then
you can just start PG from the backup and it'll perform the necessary
recovery from the WAL and start up.  We considered this an interesting
use-case and used it extensively and have maintained support for it.

All that said, of course, this limits the ability to do typical PITR,
but at least it gives you a consistent backup which you can start PG
=66rom.

Thanks,

Stephen

--3zw9Zue9ODpaeBn+
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJgIa0IAAoJEO1sijiDR2RVAe0QAJGZ471UKrWIvjTjXxaaYd1y
9WzHlVHm5GLcf0CWOngp1LC6tUsoPpslUYv951fPTkSO5vSV21qEQV2+3fskQexS
MoxATqVUrZe/HjKwOd+eE5Rr3tbb1bx7dCDSFRAnPyPqvNxyo12aAFigdAClujrc
+y2yp188LP4s3NSuggs1GP75bPTlfObv083rZNpI2Cwj706CsbSXo3KYuJKf49pB
J/Ju4fArbP1jvSuJEbhnc4e8qsdlptG/FTITSFlZj07gQpz2sMxDtTW2mPMyK0RA
W8wiuubaOw+RVvDq9sGfA7WwZHu2vxH27xF1HiSJ41/YlAsUcC2dnVQMkm2ZSuOa
JZXXZZtG1Ws8awIJw4qcceLZfwe6xQVhgb4lyZzXzlwTmiLWbrcXqH8Yhq7cLnFv
5gP1h0wrzXiFEpFAttDXWGl8gWDyDcO2n2mFRRLF4D3l5NKCo8Xvo1VD6ux5fhFi
XBE49OpWJ+LSzIK/IUiPK/HgfXofbEf4LyVdKENFeAg512kEetVryS2byIiAe20i
6dsevJbyUTEX4ZRtl5tVNFkTjUhm56ZuuPDc7ZD0QQdUEQpCvDFuMRh7bLBexplS
9D4lkZNbEvnxK9toOoGqeZiDUzt4xVDFSTPequcZNdtd9cglG/oN6ErO9oR2OF6T
yGY8VnfHPnhCvrGy3rr1
=7y2J
-----END PGP SIGNATURE-----

--3zw9Zue9ODpaeBn+--