Subject: PITR and tar

From: Jeff Davis <[email-masked]>
Date: Mon, 07 May 2007 11:58:06 -0700

The docs recommend using tar to perform a base backup for PITR.

Usually, tar reports notices like:
"tar: Truncated write; file may have grown while being archived."

First of all, is the tar archive still safe if those errors occur?

Second, it seems that it can cause a bad backup to occur if you pass the
"z" option to tar. Instead, piping the output of tar through the
compression program seems to avoid that problem (i.e. "tar cf - ... |
gzip > ..."). I am using FreeBSD's tar, other implementations may be
different.

Are my observations correct, and if so, should they be documented as a
potential "gotcha" when making base backups?

Regards,
	Jeff Davis

---

From: "Albe Laurenz" <[email-masked]>
Date: Tue, 8 May 2007 08:47:36 +0200

> "tar: Truncated write; file may have grown while being archived."

Did you call pg_start_backup(text) before you started to archive?

Yours,
Laurenz Albe

---

From: Jim Nasby <[email-masked]>
Date: Tue, 8 May 2007 10:25:58 -0500

> gzip > ..."). I am using FreeBSD's tar, other implementations may be

What *exactly* are you seeing there? If anything -z should be safer  
than piping through gzip, since you could easily accidentally pipe  
stderr through gzip as well, which *would* corrupt the backup.

> potential "gotcha" when making base backups?

I believe the bit about tar complaining about changed files is  
already in there, no?
--
Jim Nasby                                            [email-masked]
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)

---

From: Jeff Davis <[email-masked]>
Date: Tue, 08 May 2007 10:14:24 -0700

> stderr through gzip as well, which *would* corrupt the backup.

tar: Truncated write; file may have grown while being archived. 
tar: Truncated write; file may have grown while being archived. 
tar: GZip compression failed 

is the output from my cron script (which is emailed to me). This
happened several times in a row. When I tried to extract one of those
backups, I got errors like (some names have been changed):

$ tar zxf mybackup.tar.gz
data/base/16418/32309.1: Premature end of gzip compressed data:
Input/output error
tar: Premature end of gzip compressed data: Input/output error

and 
$ gzip -dc mybackup.tar.gz > /dev/null
gzip: ../mybackup.tar.gz: unexpected end of file
gzip: ../mybackup.tar.gz: uncompress failed

This may be specific to FreeBSD's tar. I remember testing in the past on
Linux and never had these problems.

When I changed to do it as a pipe instead of using the "z" flag, it
worked fine. I still get the stderr properly (which is also emailed to
me via cron) but only contains the "truncated write" warnings.

> I believe the bit about tar complaining about changed files is  

I was talking about using the "z" flag with tar causing potential bad
backups as described above, not just the warnings. If that's true, there
are probably other people with untrustworthy backups.

Regards,
	Jeff Davis

---

From: Jeff Davis <[email-masked]>
Date: Tue, 08 May 2007 10:16:50 -0700

> Did you call pg_start_backup(text) before you started to archive?

I was referring to the result of the tar itself being a corrupted gzip
file (that couldn't be uncompressed with gunzip).

I did indeed call pg_start/stop_backup(). 

Regards,
	Jeff Davis

---

From: "Merlin Moncure" <[email-masked]>
Date: Tue, 8 May 2007 13:24:28 -0400

> I did indeed call pg_start/stop_backup().

is fsync on?

merlin

---

From: Jeff Davis <[email-masked]>
Date: Tue, 08 May 2007 10:28:50 -0700

> > I did indeed call pg_start/stop_backup().

Yes. I have a battery-backed cache as well, and there were no power
failures involved. 

Regards,
	Jeff Davis

---

From: "Dhaval Shah" <[email-masked]>
Date: Wed, 9 May 2007 08:45:21 -0700

Looks like a problem specific to FreeBSD. I use Centos/postgres 8.2.3
and I do not see that problem at all.

Dhaval

On 5/8/07, Jeff Davis <[email-masked]> wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq
>

-- 
Dhaval Shah

---

From: Jim Nasby <[email-masked]>
Date: Wed, 9 May 2007 11:40:57 -0500

Actually, looking at the docs, the problem is with some versions of  
GNU tar. AFAIK bsdtar is perfectly happy to archive files that have  
changed from underneath it.

---

From: Jeff Davis <[email-masked]>
Date: Wed, 09 May 2007 10:19:05 -0700

> changed from underneath it.

$ tar --version
bsdtar 1.2.53 - libarchive 1.3.1

That fails to create a file in proper gzip format when the files are
concurrently modified.

However, 

$ tar --version
tar (GNU tar) 1.14
Copyright (C) 2004 Free Software Foundation, Inc.
This program comes with NO WARRANTY, to the extent permitted by law.
You may redistribute it under the terms of the GNU General Public
License;
see the file named COPYING for details.
Written by John Gilmore and Jay Fenlason.

That _appears_ to work.

Perhaps FreeBSD users should take notice of this problem. It's certainly
not a postgresql problem, but I know there are a lot of freebsd users
here, and using tar on fast-changing data may be rare outside of
postgresql.

Regards,
	Jeff Davis

---

From: "Jim C. Nasby" <[email-masked]>
Date: Sun, 13 May 2007 17:44:09 -0500

Moving to -docs...

Does anyone know what the history of the docs saying that GNU tar had
issues with files changing underneath it? According to this report it's
actually BSD tar that has the issue.

-- 
Jim Nasby                                      [email-masked]
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)

---

From: Bruce Momjian <[email-masked]>
Date: Sun, 13 May 2007 21:04:53 -0400 (EDT)

> actually BSD tar that has the issue.

As I remember, Tom was the one who found that GNU tar would return a
non-zero exit status if the file changed during backup, so you couldn't
determine if the backup was successful based on the exit code.

---------------------------------------------------------------------------

> TIP 4: Have you searched our list archives?
> 
>                http://archives.postgresql.org/

-- 
  Bruce Momjian  <[email-masked]>          http://momjian.us
  EnterpriseDB                               http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: Tom Lane <[email-masked]>
Date: Sun, 13 May 2007 21:46:21 -0400

> actually BSD tar that has the issue.

It seems to be a different issue.  The problem with GNU tar is that it
issues a warning and exits with nonzero status, which is a problem for
backup scripts because they can't easily distinguish this case from an
actual failure.  But AFAIK the output file is self-consistent anyway.
It sounds like bsd tar is just plain broken :-(

			regards, tom lane