Date: Tue, 12 Dec 2017 08:46:30 -0500
From: Stephen Frost <sfrost@snowman.net>
To: preeti soni <preeti_soni123@yahoo.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories
 during file system based backup
Message-ID: <20171212134630.GD4628@tamriel.snowman.net>
References: <1327412372.3442819.1513060412183.ref@mail.yahoo.com>
 <1327412372.3442819.1513060412183@mail.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="SaoeinB3Du6R2fuV"
Content-Disposition: inline
In-Reply-To: <1327412372.3442819.1513060412183@mail.yahoo.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk


--SaoeinB3Du6R2fuV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Greetings,

* preeti soni (preeti_soni123@yahoo.com) wrote:
> While performing file system based backup, after running pg_start_backup, we create a list of files in data directory called collect file.Then backup the files listed in collect file.
> This is approximately 35TB database. Very frequently we are noticing there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means, the files were present when list of files is created (Collect file). However, it is not there while backing up.
> Is the backup valid with these failed files. Any suggestion would help us.

Yes, files come and go during the backup and therefore you are likely to
have a difference between the set of files found when you create the
manifest vs. when the backup actually runs, and that should all be fine.

There's a couple different well maintained backup tools for PG which you
could (and probably should, imv) be using instead of having to guess at
these kinds of questions.

Other things that you need to be doing with a backup tool of your own is
ensuring that the WAL files created during the backup are archived and
sync'd to disk using either archive_command or through a physical
replication slot or similar (using pg_receivewal, for example).

Note that you must also verify at the end of the backup that all WAL
generated between the start backup and the stop backup have been
reliably stored.  You can determine what WAL is required based on the
information returned from the pg_start/stop_backup calls in more modern
versions of PG.  An online backup of PG isn't valid unless you have all
of that WAL.

Thanks!

Stephen

--SaoeinB3Du6R2fuV
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJaL921AAoJEO1sijiDR2RVnqQP+gI/zIYIps+RXlWmhgS38fqy
qV6fQ60dqD65sfsNwKb4NVItP4LK5dxTufALUmjT9nznnex/rd4a2QHOMpJdYYPe
vgXSpZFfe4Es1W/1gZa2S90+P021RBu9b1X795cHpnKsIAdZERtMvXfsBpQboQZZ
kAUm7GxvOhjD6yqwArzOm9Ex1oCxZQxkF3vYVcfibS656pLJcj/KkpfbhLLs3WR3
5Kp89l62ePkDqxBqxcgu0Jg1IJu2qtZDM7qM2jx+gRp2SneJH+pI8BXMrByV6cxK
t6dc+fop0iBAerk+2bn19qsb9j2iyZUMuHha38gXh2sjhBZLH+qsC867SEKs0hRV
51f4ogqMBrjbPIaKE25m/YYequrvKXBgCX86YRDDtti7rk71dmsCUkAnT1M2Qjl5
N45ACogwe4+VhUlHY3LNU+12VQJJxFkA5syLkdJ2FF+r41hlsaiOtteQRm6cRgCD
fwOdUL8lreyGM5JXLZBcuCQ8xF7breabkTGveqzewMk/QyPFrVkzYexFsCp1M+L3
jFvczItrMbNL6OAlC0bPjxydPhBnC7ShsviMcmSUdqQuJ4T4q7gayiMiwF1/hC1I
L96EqrdHoKXhhUkzZ0zZp4CqsXJ8WMUtGFvL8hl0bqm0S8wIs1+nvTp2SOFeNYcm
9jAA42ii84PAwE/nWpSF
=9m/X
-----END PGP SIGNATURE-----

--SaoeinB3Du6R2fuV--