Content-type: text/plain; charset=utf-8
MIME-version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\))
Subject: Re: Reliable WAL file shipping over unreliable network
From: Rui DeSousa <rui.desousa@icloud.com>
In-reply-to: <20180302180911.GO2416@tamriel.snowman.net>
Date: Fri, 02 Mar 2018 13:22:53 -0500
Cc: scott ribe <scott_ribe@elevated-dev.com>,
 Mark Kirkwood <mark.kirkwood@catalyst.net.nz>,
 Dianne Skoll <dfs@roaringpenguin.com>, pgsql-admin@lists.postgresql.org
Content-transfer-encoding: quoted-printable
Message-id: <BB8E0E71-D65F-409F-B9AD-DDF2237CDF99@icloud.com>
References: <20180228132620.2c882f84@hydrogen.roaringpenguin.com>
 <33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com>
 <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com>
 <5a49c387-e4f4-f70c-537c-9c338c80b687@catalyst.net.nz>
 <9DDA6F85-077E-4F8D-8708-632CC2907D3C@icloud.com>
 <06dd8b81-8b1a-ae5c-8ddd-1556b5946176@catalyst.net.nz>
 <6056907B-551F-4ED0-804B-0FCACD39EB23@icloud.com>
 <ea674584-85bc-3f63-c182-0d45123bb3eb@catalyst.net.nz>
 <988AB20C-9FDE-4197-BF3F-C54D80C2ED78@elevated-dev.com>
 <E6AB850C-D05E-405B-8D4E-DE18E128F402@icloud.com>
 <20180302180911.GO2416@tamriel.snowman.net>
To: Stephen Frost <sfrost@snowman.net>
Precedence: bulk

Not a problem; as you should be archiving your WALs to multiple sites =
simultaneously.  You=E2=80=99re DR site should not rely on any resources =
stored at the primary site.


> On Mar 2, 2018, at 1:09 PM, Stephen Frost <sfrost@snowman.net> wrote:
>=20
> Greetings,
>=20
> * Rui DeSousa (rui.desousa@icloud.com) wrote:
>>> On Mar 1, 2018, at 12:21 AM, scott ribe =
<scott_ribe@elevated-dev.com> wrote:
>>> The false report of success is not good, but it's not the root =
problem.
>>=20
>> A false success if a problem; especially in this use case as the =
source WAL file will be deleted by Postgres before it was truly =
successful.  While monitoring is nice to avoid the issue it is not a fix =
for the issue.
>>=20
>> I personally cannot recommend the use of rsync in this application =
for two reasons. =20
>=20
> There's a bigger problem that I'm amazed hasn't been brought up =
already-
> the rsync, copy, cp, whatever, may finish just fine and then the =
system
> that the WAL file was copied to crashes and you end up losing a bunch =
of
> WAL because none of these methods actually sync's the WAL to disk.
>=20
> No archive_command should ever return success until the WAL is =
actually
> written out to permanent storage and sync'd.  That's what tools like
> pgBackRest do and is part of the reason why people really shouldn't =
try
> to hack up their own archive_command solution but should be using well
> tested existing solutions.
>=20
> Thanks!
>=20
> Stephen