From: Achilleas Mantzios <achill@matrix.gatewaynet.com>
To: pgsql-admin@lists.postgresql.org
Subject: Re: Backup solution over unreliable network
Date: Fri, 30 Nov 2018 18:24:40 +0200
Message-ID: <88ba3c9e-eb96-8567-3c35-c3fdb85c1048@matrix.gatewaynet.com> (raw)
In-Reply-To: <20181130152931.GY3415@tamriel.snowman.net>
References: <fb7e7296-c60e-c2cc-93d5-9c2451e9a2a5@matrix.gatewaynet.com>
<20181130120613.GW3415@tamriel.snowman.net>
<6834c573-2a52-dd63-0264-5fc7258e7857@matrix.gatewaynet.com>
<20181130152931.GY3415@tamriel.snowman.net>
Hello Stephen!
On 30/11/18 5:29 μ.μ., Stephen Frost wrote:
> Greetings,
>
> * Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
>> On 30/11/18 2:06 μ.μ., Stephen Frost wrote:
>>>> - I have only read the best reviews about PgBackRest, can PgBackRest address those issues?
>>> Glad to hear you've read good reviews about pgbackrest. As for
>>> addressing these issues, pgbackrest has:
>>>
>>> - Backup resume
>>> - Max WAL lag (in other words, you can have it simply start throwing WAL
>>> away if it can't archive it, rather than allowing the primary to run
>>> out of disk space)
>> This is just superb! In our case we had the following architecture (now barman is defunct) :
>>
>> Primary (consistent snapshots with pg_start/stop_backup)+ --> reliable net (archive_command via rsync) --> WAL repository
>> | (async streaming replication)
>> | (reliable net)
>> V
>> Standby --> unreliable net (barman via method rsync + barman streaming from standby ***) --> remote cloud provider site(barman)
>>
>> So Primary and Standby are in the same cloud provider over consistent
>> (mostly) network, whereas the barman (remote recovery) site communicates
>> over internet. We would like to keep the old functionality (or even add a
>> new PgBackRest node in the main cloud provider, so the question is : is
>> there a way for archive-push to two different stanzas? Or delegate the
>> archive-push to work from the Standby ?
> We've had a few folks using pgbackrest to push to two stanzas by way of
> basically doing 'pgbackrest --stanza=a archive-push && pgbackrest
> --stanza=b archive-push' and with that it does work, and you could
> combine that with the max WAL setting, potentially, but it's not a
> solution that I'm really a fan of. That's definitely a use-case we've
> been thinking about though and have plans to support in the future,
> but there are other things we're tackling now and so multi-repo hasn't
> been a priority.
If we called with e.g. --archive-push-queue-max=50G for the unreliable stanza and let to default for the reliable stanza that would be OK, I guess. Yes I understand you'd like a more systematic
approach to the problem, but for the moment do you see any potential risk in doing what you described?
>
> We've also considered supporting archive-mode=always and being able to
> have the standby also push WAL and while we may support that in the
> future, I'd say it's farther down on the list than multi-repo support.
> As I recall, David Steele also had some specific technical concerns
> around how to handle two systems pushing into the same WAL archive.
I recall with barman 2.4 and postgresql 10, I had absolutely no problem receiving wal stream from the secondary and WAL archives from the primary, AS LONG AS there was no exclusive
pg_start/stop_backup happening on the primary. The moment pg_start_backup (exclusive) started on the primary, the very same barman started to complain about errors in duplicates IIRC, which means the
WALs with same name were different. The WAL themselves were identical except some bytes on the header (which I guess have to do with exclusive backup).
> Having archive-mode=always be allowed if it's going to an independent
> repo is an interesting thought though and might be simpler to do.
Yes the idea is to protect the primary and do all stuff to the secondary.
> Thanks!
Thank you!!
>
> Stephen
--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: achill@matrix.gatewaynet.com, pgsql-admin@lists.postgresql.org
Subject: Re: Backup solution over unreliable network
In-Reply-To: <88ba3c9e-eb96-8567-3c35-c3fdb85c1048@matrix.gatewaynet.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox