Date: Fri, 30 Nov 2018 07:06:13 -0500
From: Stephen Frost <sfrost@snowman.net>
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Backup solution over unreliable network
Message-ID: <20181130120613.GW3415@tamriel.snowman.net>
References: <fb7e7296-c60e-c2cc-93d5-9c2451e9a2a5@matrix.gatewaynet.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="umFY+EI/+9UScBI1"
Content-Disposition: inline
In-Reply-To: <fb7e7296-c60e-c2cc-93d5-9c2451e9a2a5@matrix.gatewaynet.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--umFY+EI/+9UScBI1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Greetings,

* Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
> we've been running our backup solution for the last 5 months to a second
> site which has an unreliable network connection. We had problems with
> barman, since it doesn't support backup resume, also no option to disable
> the replication slot, in the sense, that it is better to sacrifice the
> backup rather than fill up the primary with WALs and bring the primary down.
> Another issue is now supporting entirely backing up from the secondary. With
> barman this is not possible, streaming (or archiving) must originate from
> the primary.So I want to ask two things here :
> - Backing up to a remote site over an unreliable channel is a limited use
> case by itself, it is useful for local PITR restores on specific
> tables/data, or in case the whole primary suffers a disaster. Is there any
> other benefit that would justify building a solution for it?

Please don't build your own solution, it's really quite difficult to get
backups done correctly.

> - I have only read the best reviews about PgBackRest, can PgBackRest address those issues?

Glad to hear you've read good reviews about pgbackrest.  As for
addressing these issues, pgbackrest has:

- Backup resume
- Max WAL lag (in other words, you can have it simply start throwing WAL
  away if it can't archive it, rather than allowing the primary to run
  out of disk space)
- Backup using the replica, primairly (note that this, currently,
  requires access to the primary, but the bulk of the data comes from
  the replica)
- Incremental/differential backup
- Parallel backup/resume and parallel archiving/fetching
- Backup verification- we checksum every file backed up and verify those
  checksums on a resume, and we make sure that every WAL file needed to
  restore the backup has made it into the archive.
- Delta restore

Which I believe covers most of the use-cases you brought up.

When we first implemented backup using the replica we had concerns
regarding doing a 'full' replica-based backup, and we didn't really see
there being a lot of demand for such a use-case (the replica has access
to the primary in general if it's a streaming replica, after all...),
but we might be open to revisiting that.

Thanks!

Stephen

--umFY+EI/+9UScBI1
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJcASe0AAoJEO1sijiDR2RVWF8P/RDlO1sosYgVoQLk32EwPFHO
UUyI7JK1eMDPhDMzaNj4xP1edSsv5yFs5FHXqBJmLK4uSd6Q6YHHOB6ofLZpchh7
gOEY4mxsofex7Toq1wNZh7ZkN3101kOfM/PsmcTTs799P/g0xv4JnyDU4g1DvC2B
TtHFDxMBTL8muT0ZGRB4jSYOgguRZqEyFioNK/+VUpIhypOGT5y4y1ezufrwOoHo
rHrmtYHuRdrXhUESe2zo2X7N2Gdgt4bji/P5p11UurJtxK63BpMwsu6ql/PmdtBW
yHqTM+wmETGsliYc07glV3kWHz0yiUN+XA8tkjVCZfrmwFqwa4+BJWYD7BMhS6HC
QQXWY2gC4Grthbj+f1qVmvfIeUAknrDfhRo+grP3h7zsGYwAErQxLut+fslXxn+G
PDZVypXFsg8xPZNaDwGmvPpllBrbMfkj9IsEIqLo3x6FUomVduG36AZjcv/90xFY
C0BxPn5wGkiygVXTaOXlHQBLYDkiVJLxyE3mkvjajfG1ZxCMk9asd3ORaBfbD2xS
oiLtO3w/Wqfvv7VBytdxHPaPbvxHhFr208wbMB+JfL+BbzqsOaDQTNgFV6moS06C
HOUmZlYwtDYlwaVI7J697MqynSrxmxNlJWGjS8LZuqLVjZugQ4eqjRpx4ZiQAMOq
cI2AqBZTlNV5Yk1NOPf/
=RhAM
-----END PGP SIGNATURE-----

--umFY+EI/+9UScBI1--