Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ePMuk-0001r3-3C for pgsql-admin@arkaria.postgresql.org; Thu, 14 Dec 2017 06:22:38 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ePMui-0000lp-DV for pgsql-admin@arkaria.postgresql.org; Thu, 14 Dec 2017 06:22:36 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1ePMui-0000lg-47 for pgsql-admin@lists.postgresql.org; Thu, 14 Dec 2017 06:22:36 +0000 Received: from sonic316-22.consmr.mail.ne1.yahoo.com ([66.163.187.148]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1ePMue-0003m5-VN for pgsql-admin@lists.postgresql.org; Thu, 14 Dec 2017 06:22:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1513232550; bh=+UFs8MkIJPqO2ljBDz/s7tr51FDWRWHHMGpjRqCVpQc=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject; b=l+e5SOlZMn5Q8Q8oWpt+6KfYV/LaRU/vMPYo9eN8V0WhVYNJm9ozkEtUFoHmljclZP+a02xJgV6G5tEd2x5rixmV0/CC517t8LVN+oa53z7UEd0YTnU3FBtsDB7BckuEAKyxCffBaFnusTxzvmA2M824aPEbRTvpBkKga5gIn4H/u9rDn3/qia1kJK0sLSGpKoAR/8a6P/YI7oF4CPztfCBaYThJx3729EKhiysgfOkc6WLo2jdpHHHQRXmWjN0rqDgtLq+hQuUvxD/i5XznPyhw+MazPBVyO5b1KGVPNXkygt63ofeH7zQNC8ci4/cm01FNh7shNdQJbbXa3GZeLg== X-YMail-OSG: xRYDsk4VM1knKtrAS.e_Ko7EVMQqFig3ZX2RG89HR7qszVQXxpC9fHlqLc8BML6 oPJ2Uqryve4DAYs0GiTURNkKHi9S4YQwsyBpNGPZY1dO9cd4tITTlMzV0DaXuAXisL4FET.ImrRW SVnx5499pdU4EJjiMhKpxX.4ZZ_llAkjbmFWRyklxw2MNO9FH9gvzg9Evzd0TFFEeos0UmrlAW3O um7xE6Y5F9EWHhTBoeBy5y1jgWF3x3VjJxAHwBaOtIq_gIX3FL1YGYk6dOV2McWXVYd0MEKbTdzJ MJY71nLmsYg_GUuY6K2XPkrqQOUnn.zuKBCrLHYyt45cQElLbcPgPFMYVPruk41qOTr8TW22T8bX tZcgZRIEDM0BTtFwA91iJjjfNRFrMgrvKW.zklXFgIlEHdZZkJ2UhVthRvtWyGsrINFgHlO_nfQQ shs3VFj_WfljjM3esbPwtyxSm68eRBPhDSQSer_f.aItG66GLNRlRIEaSbRXt9p1yT_fGs.MNxgT lPaOedXL1gebM Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.ne1.yahoo.com with HTTP; Thu, 14 Dec 2017 06:22:30 +0000 Date: Thu, 14 Dec 2017 06:22:11 +0000 (UTC) From: preeti soni To: Stephen Frost Cc: pgsql-admin@lists.postgresql.org Message-ID: <2027432855.5222226.1513232531507@mail.yahoo.com> In-Reply-To: <20171212134630.GD4628@tamriel.snowman.net> References: <1327412372.3442819.1513060412183.ref@mail.yahoo.com> <1327412372.3442819.1513060412183@mail.yahoo.com> <20171212134630.GD4628@tamriel.snowman.net> Subject: Re: Missing files in pg_xlog, pg_subtrans and pg_clog directories during file system based backup MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_5222225_1869467437.1513232531505" X-Mailer: WebService/1.1.11051 YMailNorrin Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36 Content-Length: 6106 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk ------=_Part_5222225_1869467437.1513232531505 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks a lot Stephen! We are using Commvault's solution for File system based backup. pg_start_ba= ckup/pg_stop_backup is executed to get the start log number and end log num= ber. Also, the required WAL files are successfully backed up along with dat= a backup. So it means, backup is valid in this case, with missing files in pg_xlog, p= g_subtrans and pg_clog. Thanks,Preeti On Tuesday, December 12, 2017, 7:16:58 PM GMT+5:30, Stephe= n Frost wrote: =20 =20 Greetings, * preeti soni (preeti_soni123@yahoo.com) wrote: > While performing file system based backup, after running pg_start_backup,= we create a list of files in data directory called collect file.Then backu= p the files listed in collect file. > This is approximately 35TB database. Very frequently we are noticing ther= e are failed files in pg_xlog, pg_subtrans and pg_clog directory. It means,= the files were present when list of files is created (Collect file). Howev= er, it is not there while backing up. > Is the backup valid with these failed files. Any suggestion would help us= . Yes, files come and go during the backup and therefore you are likely to have a difference between the set of files found when you create the manifest vs. when the backup actually runs, and that should all be fine. There's a couple different well maintained backup tools for PG which you could (and probably should, imv) be using instead of having to guess at these kinds of questions. Other things that you need to be doing with a backup tool of your own is ensuring that the WAL files created during the backup are archived and sync'd to disk using either archive_command or through a physical replication slot or similar (using pg_receivewal, for example). Note that you must also verify at the end of the backup that all WAL generated between the start backup and the stop backup have been reliably stored.=C2=A0 You can determine what WAL is required based on the information returned from the pg_start/stop_backup calls in more modern versions of PG.=C2=A0 An online backup of PG isn't valid unless you have al= l of that WAL. Thanks! Stephen =20 ------=_Part_5222225_1869467437.1513232531505 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks a lot Stephen!

We are usin= g Commvault's solution for File system based backup. pg_start_backup/pg_sto= p_backup is executed to get the start log number and end log number. Also, = the required WAL files are successfully backed up along with data backup.

So it means, backup is valid in this case, with mis= sing files in pg_xlog, pg_subtrans and pg_clog.

Thanks,
Preeti
=20
=20
On Tuesday, December 12, 2017, 7:16:58 PM GMT+5:30,= Stephen Frost <sfrost@snowman.net> wrote:


Greetings,

* = preeti soni (preeti_soni123@yahoo.com) wrot= e:
> While performing file system based backup, after = running pg_start_backup, we create a list of files in data directory called= collect file.Then backup the files listed in collect file.
> This is approximately 35TB database. Very frequently we are noticin= g there are failed files in pg_xlog, pg_subtrans and pg_clog directory. It = means, the files were present when list of files is created (Collect file).= However, it is not there while backing up.
> Is the b= ackup valid with these failed files. Any suggestion would help us.

Yes, files come and go during the backup and t= herefore you are likely to
have a difference between the = set of files found when you create the
manifest vs. when = the backup actually runs, and that should all be fine.
There's a couple different well maintained backup tools fo= r PG which you
could (and probably should, imv) be using = instead of having to guess at
these kinds of questions.
Other things that you need to be doing = with a backup tool of your own is
ensuring that the WAL f= iles created during the backup are archived and
sync'd to= disk using either archive_command or through a physical
= replication slot or similar (using pg_receivewal, for example).

Note that you must also verify at the end of the = backup that all WAL
generated between the start backup an= d the stop backup have been
reliably stored.  You ca= n determine what WAL is required based on the
information= returned from the pg_start/stop_backup calls in more modern
versions of PG.  An online backup of PG isn't valid unless you hav= e all
of that WAL.


Thanks!


Stephen

------=_Part_5222225_1869467437.1513232531505--