Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup

pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed

Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
5+ messages / 3 participants
[nested] [flat]

* Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
@ 2017-12-12 06:33  preeti soni <preeti_soni123@yahoo.com>
  0 siblings, 2 replies; 5+ messages in thread

From: preeti soni @ 2017-12-12 06:33 UTC (permalink / raw)
  To: pgsql-admin@lists.postgresql.org

Hi,
While performing file system based backup, after running pg_start_backup, we create a list of files in data directory called collect file.Then backup the files listed in collect file.
This is approximately 35TB database. Very frequently we are noticing there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means, the files were present when list of files is created (Collect file). However, it is not there while backing up.
Is the backup valid with these failed files. Any suggestion would help us.

Thanks in advance!!
Regards,Preeti

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
@ 2017-12-12 07:20  Shreeyansh Dba <shreeyansh2014@gmail.com>
  parent: preeti soni <preeti_soni123@yahoo.com>
  1 sibling, 1 reply; 5+ messages in thread

From: Shreeyansh Dba @ 2017-12-12 07:20 UTC (permalink / raw)
  To: preeti soni <preeti_soni123@yahoo.com>; +Cc: pgsql-admin@lists.postgresql.org

On Tue, Dec 12, 2017 at 12:03 PM, preeti soni <preeti_soni123@yahoo.com>
wrote:

> Hi,
>
> While performing file system based backup, after running pg_start_backup,
> we create a list of files in data directory called collect file.
> Then backup the files listed in collect file.
>
> This is approximately 35TB database. Very frequently we are noticing there
> are failed files in
> 
> pg_xlog, pg_subtrans and pg_clog directory. It means, the files were
> present when list of files is created (Collect file). However, it is not
> there while backing up.
>
> Is the backup valid with these failed files. Any suggestion would help us.
>

No, the backup with missing files in

pg_xlog, pg_subtrans and pg_clog directory
Please do check your server logs for more details where the error is
occurring actually

>
> Thanks in advance!!
>
> Regards,
> Preeti
>
>


-- 
[image: http://www.shreeyansh.com] <http://www.shreeyansh.com;

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
@ 2017-12-12 09:54  preeti soni <preeti_soni123@yahoo.com>
  parent: Shreeyansh Dba <shreeyansh2014@gmail.com>
  0 siblings, 0 replies; 5+ messages in thread

From: preeti soni @ 2017-12-12 09:54 UTC (permalink / raw)
  To: Shreeyansh Dba <shreeyansh2014@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org

 Thanks Shreeyansh for prompt response!!
I am checking server logs. Will update the thread with findings.

Thanks,Preeti
    On Tuesday, December 12, 2017, 12:50:56 PM GMT+5:30, Shreeyansh Dba <shreeyansh2014@gmail.com> wrote:  

On Tue, Dec 12, 2017 at 12:03 PM, preeti soni <preeti_soni123@yahoo.com> wrote:

Hi,
While performing file system based backup, after running pg_start_backup, we create a list of files in data directory called collect file.Then backup the files listed in collect file.
This is approximately 35TB database. Very frequently we are noticing there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means, the files were present when list of files is created (Collect file). However, it is not there while backing up.
Is the backup valid with these failed files. Any suggestion would help us.

No, the backup with missing files in pg_xlog, pg_subtrans and pg_clog directory
Please do check your server logs for more details where the error is occurring actually

Thanks in advance!!
Regards,Preeti

-- 

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
@ 2017-12-12 13:46  Stephen Frost <sfrost@snowman.net>
  parent: preeti soni <preeti_soni123@yahoo.com>
  1 sibling, 1 reply; 5+ messages in thread

From: Stephen Frost @ 2017-12-12 13:46 UTC (permalink / raw)
  To: preeti soni <preeti_soni123@yahoo.com>; +Cc: pgsql-admin@lists.postgresql.org

Greetings,

* preeti soni (preeti_soni123@yahoo.com) wrote:
> While performing file system based backup, after running pg_start_backup, we create a list of files in data directory called collect file.Then backup the files listed in collect file.
> This is approximately 35TB database. Very frequently we are noticing there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means, the files were present when list of files is created (Collect file). However, it is not there while backing up.
> Is the backup valid with these failed files. Any suggestion would help us.

Yes, files come and go during the backup and therefore you are likely to
have a difference between the set of files found when you create the
manifest vs. when the backup actually runs, and that should all be fine.

There's a couple different well maintained backup tools for PG which you
could (and probably should, imv) be using instead of having to guess at
these kinds of questions.

Other things that you need to be doing with a backup tool of your own is
ensuring that the WAL files created during the backup are archived and
sync'd to disk using either archive_command or through a physical
replication slot or similar (using pg_receivewal, for example).

Note that you must also verify at the end of the backup that all WAL
generated between the start backup and the stop backup have been
reliably stored.  You can determine what WAL is required based on the
information returned from the pg_start/stop_backup calls in more modern
versions of PG.  An online backup of PG isn't valid unless you have all
of that WAL.

Thanks!

Stephen

Attachments:

  signature.asc (application/pgp-signature, 818B) download

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup
@ 2017-12-14 06:22  preeti soni <preeti_soni123@yahoo.com>
  parent: Stephen Frost <sfrost@snowman.net>
  0 siblings, 0 replies; 5+ messages in thread

From: preeti soni @ 2017-12-14 06:22 UTC (permalink / raw)
  To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-admin@lists.postgresql.org

 Thanks a lot Stephen!
We are using Commvault's solution for File system based backup. pg_start_backup/pg_stop_backup is executed to get the start log number and end log number. Also, the required WAL files are successfully backed up along with data backup.
So it means, backup is valid in this case, with missing files in pg_xlog, pg_subtrans and pg_clog.

Thanks,Preeti    On Tuesday, December 12, 2017, 7:16:58 PM GMT+5:30, Stephen Frost <sfrost@snowman.net> wrote:  

 Greetings,

* preeti soni (preeti_soni123@yahoo.com) wrote:
> While performing file system based backup, after running pg_start_backup, we create a list of files in data directory called collect file.Then backup the files listed in collect file.
> This is approximately 35TB database. Very frequently we are noticing there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means, the files were present when list of files is created (Collect file). However, it is not there while backing up.
> Is the backup valid with these failed files. Any suggestion would help us.

Yes, files come and go during the backup and therefore you are likely to
have a difference between the set of files found when you create the
manifest vs. when the backup actually runs, and that should all be fine.

There's a couple different well maintained backup tools for PG which you
could (and probably should, imv) be using instead of having to guess at
these kinds of questions.

Other things that you need to be doing with a backup tool of your own is
ensuring that the WAL files created during the backup are archived and
sync'd to disk using either archive_command or through a physical
replication slot or similar (using pg_receivewal, for example).

Note that you must also verify at the end of the backup that all WAL
generated between the start backup and the stop backup have been
reliably stored.  You can determine what WAL is required based on the
information returned from the pg_start/stop_backup calls in more modern
versions of PG.  An online backup of PG isn't valid unless you have all
of that WAL.

Thanks!

Stephen

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

end of thread, other threads:[~2017-12-14 06:22 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-12-12 06:33 Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup preeti soni <preeti_soni123@yahoo.com>
2017-12-12 07:20 ` Shreeyansh Dba <shreeyansh2014@gmail.com>
2017-12-12 09:54   ` preeti soni <preeti_soni123@yahoo.com>
2017-12-12 13:46 ` Stephen Frost <sfrost@snowman.net>
2017-12-14 06:22   ` preeti soni <preeti_soni123@yahoo.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox