Date: Fri, 2 Mar 2018 13:09:11 -0500
From: Stephen Frost <sfrost@snowman.net>
To: Rui DeSousa <rui.desousa@icloud.com>
Cc: scott ribe <scott_ribe@elevated-dev.com>,
	Mark Kirkwood <mark.kirkwood@catalyst.net.nz>,
	Dianne Skoll <dfs@roaringpenguin.com>,
	pgsql-admin@lists.postgresql.org
Subject: Re: Reliable WAL file shipping over unreliable network
Message-ID: <20180302180911.GO2416@tamriel.snowman.net>
References: <20180228132620.2c882f84@hydrogen.roaringpenguin.com>
 <33220469-5BB2-4A40-8F56-8F4F392C95B3@icloud.com>
 <21AF0CB4-0873-4960-BA14-E72FA08B352E@icloud.com>
 <5a49c387-e4f4-f70c-537c-9c338c80b687@catalyst.net.nz>
 <9DDA6F85-077E-4F8D-8708-632CC2907D3C@icloud.com>
 <06dd8b81-8b1a-ae5c-8ddd-1556b5946176@catalyst.net.nz>
 <6056907B-551F-4ED0-804B-0FCACD39EB23@icloud.com>
 <ea674584-85bc-3f63-c182-0d45123bb3eb@catalyst.net.nz>
 <988AB20C-9FDE-4197-BF3F-C54D80C2ED78@elevated-dev.com>
 <E6AB850C-D05E-405B-8D4E-DE18E128F402@icloud.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="7xAWiQHeXYvug0yh"
Content-Disposition: inline
In-Reply-To: <E6AB850C-D05E-405B-8D4E-DE18E128F402@icloud.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--7xAWiQHeXYvug0yh
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Rui DeSousa (rui.desousa@icloud.com) wrote:
> > On Mar 1, 2018, at 12:21 AM, scott ribe <scott_ribe@elevated-dev.com> w=
rote:
> > The false report of success is not good, but it's not the root problem.
>=20
> A false success if a problem; especially in this use case as the source W=
AL file will be deleted by Postgres before it was truly successful.  While =
monitoring is nice to avoid the issue it is not a fix for the issue.
>=20
> I personally cannot recommend the use of rsync in this application for tw=
o reasons. =20

There's a bigger problem that I'm amazed hasn't been brought up already-
the rsync, copy, cp, whatever, may finish just fine and then the system
that the WAL file was copied to crashes and you end up losing a bunch of
WAL because none of these methods actually sync's the WAL to disk.

No archive_command should ever return success until the WAL is actually
written out to permanent storage and sync'd.  That's what tools like
pgBackRest do and is part of the reason why people really shouldn't try
to hack up their own archive_command solution but should be using well
tested existing solutions.

Thanks!

Stephen

--7xAWiQHeXYvug0yh
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJamZNHAAoJEO1sijiDR2RVCeoP+wcCw6XXu59Sc7P8iBRzJmMs
t8RK3+3ga2D4gqqD4J2Yzf1DA2N+mlnwvA3ghG3H3VE6R0C8DCVWTBIe16ILnidO
J75LP4A2soGyiLMBDpHbJ17QZBTuhiYkSMWU00vXk+dWahxwxUU42d97/ZnX5oeA
QkHnV25cSjRd9zIW/UnNMacgpIJS0c44kpIu1ZUFnN0C9nH4B/Ov/rwiTQoETWb1
gAgTxSG/5H6nokPfkwSyNQ6nlF1OHfU5AXIbYCQRfTQ1KZdkOi73uvIA2ZTrTBZ5
G8f2dWb4HdA+BaIyKT5Xeoj3Mpe3giMqSyTRSg1QSyfbvfP9+B7Yg1PY8vmEYSzz
+mnnduwZSCwL4klD9Z0YPT6jLhTfP7faZX3kpdvk1L8otNbJ514Fy4FgaRHl3vRh
vfaH34kDzjqcso0FzmEqv0fw6NR5qt+JUtZ4x7VKThrmX/jk9FayB8sNakDw0JTD
kAOMvJ+07PO7po4IFdHVju+AeCFERISc85Jc70+/XYvPizT7ENtaTljxcF/1Iymh
lHmGmFQnezkZd3z4q+A956pn/saAeol69/hZJ99S9kgvQJJksueOBd2sw424hEW7
Mx8F80QKXw+S1WPF3s5bhWBEw2iMoLfhcUyGE+PALVduQ4IvEPiC9rQtHkmmfvzm
/qt6Jyux4fDGEL8G1niM
=p79Y
-----END PGP SIGNATURE-----

--7xAWiQHeXYvug0yh--