Subject: Re: Backup solution over unreliable network
To: pgsql-admin@lists.postgresql.org
References: <fb7e7296-c60e-c2cc-93d5-9c2451e9a2a5@matrix.gatewaynet.com>
 <20801752-C311-4F70-9D03-08C5E711645C@mac.com>
From: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Message-ID: <9956da38-bcb6-6768-d245-887df9e0fb93@matrix.gatewaynet.com>
Date: Fri, 30 Nov 2018 20:49:39 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
In-Reply-To: <20801752-C311-4F70-9D03-08C5E711645C@mac.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk


On 30/11/18 8:22 μ.μ., Evan Bauer wrote:
> Achilleas,
>
> I may be over-simplifying your situation, but have you considered breaking the problem into two pieces?  First backing up to a local drive and then using rsync to move those files over the unreliable network to the remote site.
>
> Like other who have responded, I can heartily recommend pgbackrest.  But if the network stinks, then I’d break the problem in two and leave PostgreSQL out of the network equation.


Pretty good idea, but :

1) those rsync transfers have to be somehow db aware, otherwise lots of 
things might break, checksums, order of WALs, etc. There would be the 
need to write a whole solution and end up ... getting one of the 
established solutions

2) the rsync part would go basically unattended, meaning no smart 
software would be taking care of it, monitoring it, sending alerts, etc. 
Also we had our issues with rsync in the past with unreliable networks 
like getting error messages for which google returns one or no results 
(no pgsql stuff, just system scripts) . No wonder more and more PgSQL 
backup solutions move away from rsync.

Am I missing something or exaggerating?


>
> Cheers,
>
> Evan
>
> Sent from my iPhone
>
>> On Nov 30, 2018, at 05:17, Achilleas Mantzios <achill@matrix.gatewaynet.com> wrote:
>>
>> Hello,
>> we've been running our backup solution for the last 5 months to a second site which has an unreliable network connection. We had problems with barman, since it doesn't support backup resume, also no option to disable the replication slot, in the sense, that it is better to sacrifice the backup rather than fill up the primary with WALs and bring the primary down. Another issue is now supporting entirely backing up from the secondary. With barman this is not possible, streaming (or archiving) must originate from the primary.So I want to ask two things here :
>> - Backing up to a remote site over an unreliable channel is a limited use case by itself, it is useful for local PITR restores on specific tables/data, or in case the whole primary suffers a disaster. Is there any other benefit that would justify building a solution for it?
>> - I have only read the best reviews about PgBackRest, can PgBackRest address those issues?
>>
>> Thank you!
>>
>> -- 
>> Achilleas Mantzios
>> IT DEV Lead
>> IT DEPT
>> Dynacom Tankers Mgmt
>>
>>
>