Date: Tue, 29 Nov 2016 08:43:23 -0500
From: Stephen Frost <sfrost@snowman.net>
To: "Gunnar \"Nick\" Bluth" <gunnar.bluth@pro-open.de>
Cc: "pgsql-docs@postgresql.org" <pgsql-docs@postgresql.org>,
	Kevin Grittner <kgrittn@gmail.com>
Subject: Re: Missing important information in backup.sgml
Message-ID: <20161129134323.GO13284@tamriel.snowman.net>
References: <cb648ec1-5212-60c5-b73a-9572d21b2ae9@elster.de>
 <cec57bd3-3742-9bae-a76f-f3414dcf135e@elster.de>
 <20161116143651.GC13284@tamriel.snowman.net>
 <80e18b98-9b72-ddee-2a34-76302e5d8a0b@pro-open.de>
 <cd880db6-1ff6-8fa7-4868-249421cd6fb2@pro-open.de>
 <CACjxUsMAoDNAMvE_bHpb0OzoamVDk93fESGMOfv_aR6E-Q6vUg@mail.gmail.com>
 <a5a614b8-9ce2-ae3b-1141-c09e391cbba2@pro-open.de>
 <20161123204139.GZ13284@tamriel.snowman.net>
 <18630438-5cf7-ae42-fc51-30770fb43f54@pro-open.de>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="2kPS9oLXDLT6LUTO"
Content-Disposition: inline
In-Reply-To: <18630438-5cf7-ae42-fc51-30770fb43f54@pro-open.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk
Sender: pgsql-docs-owner@postgresql.org


--2kPS9oLXDLT6LUTO
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Gunnar "Nick" Bluth (gunnar.bluth@pro-open.de) wrote:
> Am 23.11.2016 um 21:41 schrieb Stephen Frost:
> > * Gunnar "Nick" Bluth (gunnar.bluth@pro-open.de) wrote:
> > One of the very important things that should be done as part of a backup
> > is to ensure that all of the archive files required to restore the
> > database to a consistent state are safely stored in the archive.  If
> > that isn't done then it's possible that an incomplete archive may also
> > render backups invalid.
>=20
> Well, the need to have a complete archive is described in the docs
> already. Maybe the potential consequences of an incomplete archive
> should be pointed or more drastically...?

We should probably add to "step 5" something about "verify that all WAL
files have been archived between the start and stop backup" or similar.

> Now, the main purpose of my patch was to document a behaviour that many
> of us have run into, namely that FATAL error showing up in the log when
> the archive_command exits with RC > 127. It's a nuisance only, but it
> does send people on false tracks and should at least be mentioned in the
> documentation.

I agree that we should add information to the documentation that certain
error codes are handled differently.

> And since a couple of people does use rsync (or some wrappers around it)
> for archiving, and that is notoriously giving RCs > 127, it seems legit
> to at least mention it, no?

The low-level API documentation should be focused on the API and not how
to (mis)use the API using common unix commands.

> diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
> index 6eaed1e..a8f574e 100644
> --- a/doc/src/sgml/backup.sgml
> +++ b/doc/src/sgml/backup.sgml
> @@ -587,7 +587,8 @@ tar -cf backup.tar /usr/local/pgsql/data
>      the administrator specify a shell command to be executed to copy a
>      completed segment file to wherever it needs to go.  The command coul=
d be
>      as simple as a <literal>cp</>, or it could invoke a complex shell
> -    script &mdash; it's all up to you.
> +    script &mdash; it's all up to you. There however are some things to =
consider
> +    when creating such a command, most of which are covered below.
>     </para>

If we're changing this then we should remove the notion that it could be
"as simple as a cp" because it really should *not* be.  That is a
dangerous and very poor recommendation to be making to our users.

>     <para>
> @@ -636,7 +637,11 @@ test ! -f /mnt/server/archivedir/00000001000000A9000=
00065 &amp;&amp; cp pg_wal/0
>      <productname>PostgreSQL</> will assume that the file has been
>      successfully archived, and will remove or recycle it.  However, a no=
nzero
>      status tells <productname>PostgreSQL</> that the file was not archiv=
ed;
> -    it will try again periodically until it succeeds.
> +    it will try again periodically until it succeeds. Note that an exit
> +    status of 128 or higher will cause the archiver to exit, resulting i=
n a
> +    <literal>FATAL</> error in the server log. It will be restarted by t=
he
> +    postmaster and continue where it left. E.g., <command>rsync</> is kn=
own
> +    for returning exit statuses of 255 on network issues.
>     </para>

I agree with adding documentation about what happens with different exit
status values.  I don't believe we should make any mention of rsync.  A
sophisticated enough user to understand exit codes should be able to
work out what the exit code is for whatever tool or tools they're using,
it's not on us to document what the exit codes are for every possible
tool (nor should one even use something as simple as a bare cp or rsync
in archive_command).

>     <para>
> @@ -696,6 +701,16 @@ test ! -f /mnt/server/archivedir/00000001000000A9000=
00065 &amp;&amp; cp pg_wal/0
>      preserve the file name (<literal>%f</>).
>     </para>
> =20
> +  <para>
> +    Depending on your specific requirements (and datacenter layout), you=
 may
> +    want to make sure that the archived WAL segments have been written o=
ut to
> +    persistent storage before the <command>archive_command</> returns.
> +    Otherwise, a WAL segment that is assumed by <productname>PostgreSQL<=
/>
> +    to be archived could be recycled or removed prematurely, rendering y=
our
> +    archive incomplete (and thus disabling a recovery) in case of an out=
age of
> +    the archiving destination.
> +  </para>

I dislike the tone of this.  The wording I would suggest would be more
along the lines of:

The archive_command should only return success once the WAL segment has
been completely copied, written out, and synced to persistent storage as
PostgreSQL will recycle or remove the segment shortly after the
archive_command returns.  If the WAL segment is lost due to an outage of
the archive server or other issue, any backup which was performed during
the time that the WAL segment was written will be unusable and
Point-In-Time-Recovery from an earlier backup will not be usable past
the missing WAL segment.

Thanks!

Stephen

--2kPS9oLXDLT6LUTO
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJYPYX7AAoJEO1sijiDR2RVTeAQALyXw/lKSGmupNlJN1WjWjf3
/uByQUnzS2HTOn6i7KA/2+o9uxsIJ/CHUnkWljFXJZHTpKL21mXVkjy80Vc7SfBE
/RlNOqjzi61vjA8+9LVg7Pt3OWbplwANuhIbu9uVh/1+HyGX9FRtvEgjkbEw9tvq
FZmBLweZJxja0ga2OvaNRhxgyxYb0ryGM2VNDWuxDK8pyjNaaEXha8wIypcJR517
760RydUCmz3Pkxm2JneO90qCeq7s38u2NXIqvnMosAIBp8GuozsRX5DJYEoIhoGi
gnzA76l2wDkYLuPu3hV/zHX0CBsbPsnlor/XMIc1SAh0azD57G9ZPIzcOfH1/9nf
j2HeihDp3ZoA0fUZgPQ2FoeiDEvZsoLpjEqJMF1o+Tw7OvWW2DWZCOmAjP9/yvoC
4gLc377ezjSjgQc4KvR+GGE+g/67Tb7EvN2fCmGhxKa+C1JcCufHitP0P1l9DiM2
UKPksfkYZ/zkesF47mYcPHd1oXpJ4xck4ol1LOsYjRXUqFX3zromZXqa4REHZiiG
BAvXjnrb118dl0P9B+wFWLyWLbX6dGcFXB4iK5yL+CloohKSXqO4GFNqq0aRNdqo
FFjjb5QD0pRk48X52gym9H0QZCBgPal1es3EMW70ipSVK9PY+5hqZg0uM7rwDMI7
objwckHqQZoiM8qypjB7
=cIHy
-----END PGP SIGNATURE-----

--2kPS9oLXDLT6LUTO--