Date: Tue, 31 Oct 2017 07:47:56 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Cc: pgsql-admin@postgresql.org
Subject: Re: Bad recovery: no pg_xlog/RECOVERYXLOG
Message-ID: <20171031114756.GP4628@tamriel.snowman.net>
References: <d90474b1-6913-f3ff-d224-73fd931cdd98@cahoots.pl>
 <1070071508952614@web46j.yandex.ru>
 <ae7f2ee9-97ff-267e-b6dc-9399250a9314@cahoots.pl>
 <20171030154706.GE4628@tamriel.snowman.net>
 <7966ee9e-2241-eced-4639-e32c157b915a@catalyst.net.nz>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="5Z7NaoLe0vff9JhZ"
Content-Disposition: inline
In-Reply-To: <7966ee9e-2241-eced-4639-e32c157b915a@catalyst.net.nz>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org


--5Z7NaoLe0vff9JhZ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Mark,

* Mark Kirkwood (mark.kirkwood@catalyst.net.nz) wrote:
> On 31/10/17 04:47, Stephen Frost wrote:
> >* Marcin Koziej (marcin@cahoots.pl) wrote:
> >>Now it's fixed, but if anyone needs I'm attaching all scripts to 1)
> >>backup and restore wal's and 2) backup and restore base backup from
> >>OpenStack SWIFT
> >Interesting, but these scripts seem to be seriously lacking in error
> >checking (what happens if the copy to swift fails..?  or pg_basebackup
> >fails?) and it's unclear how you can be sure that the WAL file has been
> >sync'd to disk which is important or you might end up having holes in
> >your WAL stream if the swift system fails.  There's also no checking to
> >make sure that the WAL needed for a given pg_basebackup ever actually
> >made it to the swift system, which is required to ensure you have a
> >consistent backup.
> >
> >Generally speaking, these kinds of scripts really aren't a good choice
> >for doing backups of PG.  I'd strongly suggest you look at one of the
> >existing tools which are developed specifically for doing backups of PG
> >and are well tested, supported, and maintained.  If you'd like support
> >for a new storage system, I know that at least pgBackRest's storage
> >layer is pluggable and adding a new storage option is pretty straight
> >forward.
>
> I'm not convinced that his approach is bad.

I was the same way for a long time, thinking that shell scripts could
reasonably be used with certain caveats, but the devil really is in the
details and it's far too easy to miss things in shell scripts (such as
not checking return codes, or not doing so properly, or various other
issues).  Also, you didn't address things like verifying that you
actually have all the WAL needed for a valid backup, and how to handle
retention?

> The script checks the result of the 'swift upload' for the base
> backup, it is the wal backup one that does not explicitly check the
> 'swift upload' result (this should really be added). To be fair,
> anything wrong with the swift system will likely be discovered
> immediately beforehand where he does a 'swift stat'!

Things could certainly break between those two calls to swift, in a
variety of ways.

> I'd guess his original problem was an improperly setup
> recovery.conf, rather than the overall design.

I agree that the original issue is unlikely to be related to these
scripts.  That doesn't mean that using them is a good idea.

Thanks!

Stephen

--5Z7NaoLe0vff9JhZ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJZ+GLsAAoJEO1sijiDR2RV6YoP/2S/6Cn7UekcJVHqiwCTwvwK
xWfqCqLac3sNWvUxwzU7Bk4cJrTLZ5eHROfTta9lJbBLWyvGqs0uKok+vrwhk3Dy
7C1IOTJEtvNiB8MzvzUTF9h4fp0VXF6QE2aCCV7qvtjHVxhpfTvMwvE9ZjnPrPvr
jtu2PjrGWLgtgoHgVC7gm7JxRAylbTG/vcnz1bcXC5bfdkOeU+xG+HDAJltpQ9eh
1KZAnPJvIQxRZxsNv+aWqUGFJ9Xg2UHj2pOztBjzrsxC48gdfWjmOELQOEyzruZX
2Ws1m7GVrUiG5F6i4W+lGS3MdAkkVMzVbyxllyOjmUUjaJEOXp7MC8ySyx5QBueG
5eEJ4yIlcm3MlC5mYKgabgi+6RXjOIMws75Exak2rwzr0w1GlHq7qy4ytt0qD9o2
zuTTFIg4Ir8UQScuykHedx5fNqlAUSeZmjN9HIx3nWr+2XnSKQxuazKIwBbNz5If
XF8e8Vmn0E7iyoxBkcMTH37dDUjzZxFZYc5b+yY/wKHU+j273qNx6X217g1ZfSmK
3N0fRn75h9niJCypJrhvGTe5ZP2ZKSNVw8msruaJLbDzwskVGouLGIg5IhKo59Xb
xykVETc71deuXAouyiQW/X/h2rh1p93ZS50MKIvPBO3dePl37S4VZwXlGdJuhJ+m
0ZnqZoy/HtSxtqApNZXQ
=h/M0
-----END PGP SIGNATURE-----

--5Z7NaoLe0vff9JhZ--