Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XV0Vv-0001YB-NR for pgsql-admin@arkaria.postgresql.org; Fri, 19 Sep 2014 15:54:27 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XV0Vu-0008Fb-KA for pgsql-admin@arkaria.postgresql.org; Fri, 19 Sep 2014 15:54:26 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XV0Vd-0007ky-Ij for pgsql-admin@postgresql.org; Fri, 19 Sep 2014 15:54:09 +0000 Received: from mail-ie0-f179.google.com ([209.85.223.179]) by makus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XV0VW-0001Br-1W for pgsql-admin@postgresql.org; Fri, 19 Sep 2014 15:54:05 +0000 Received: by mail-ie0-f179.google.com with SMTP id rl12so3859504iec.10 for ; Fri, 19 Sep 2014 08:53:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version:content-type; bh=md6OeYFzveZwGXo2RaUpZDUDBFTJWYgBoMyiRI8KzgA=; b=ilYyqnnKWMUN+6cJcqgdaRFz8tLJkmupYN7hiMIm7IQ+80QC8RufmMZyz0FWfqwhMU VmZrSaMboY6nSLBZApvbmyQwRPq38bzVbp0HOcy38s1LiYv9CJJGUoIxymU8a7m/kOsM 3OM0W4HlEoyGn6dVBChb18/+sUiCEs1YWOqtfkKqJLMq1E/TTgV4o8kQaxAlvLsL/0FC cxDuuT6iLppIGSqTSj4EGP631erY6zn1KWGiXepL5uNyneA6W7HB+W7oUdtOchqk8D9T 03CUirAV2S2LCCMCXHxsv1WMbRhXa4YVCyeGlIB3DOsYFKwL4G0QXS1eAVBUFEJ2b32R 1QaA== X-Gm-Message-State: ALoCoQlJVQ653MqCoZCl/M2ulkkaem+ux6O3aBNdcHivNDOAd+gojA4Lo+DWWMdSIydh8cwtw6669aG/lNSATHjGA2I1idXa3dZG6IyweyAbDVsJ9JVx9shjfrBYmtKUvNwjnsiZH+rB X-Received: by 10.50.43.138 with SMTP id w10mr8134195igl.33.1411142038379; Fri, 19 Sep 2014 08:53:58 -0700 (PDT) Received: from palos ([50.76.140.65]) by mx.google.com with ESMTPSA id l9sm1902570igf.9.2014.09.19.08.53.56 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 19 Sep 2014 08:53:57 -0700 (PDT) Date: Fri, 19 Sep 2014 09:53:55 -0600 From: Jason Mathis To: "=?utf-8?Q?pgsql-admin=40postgresql.org?=" , =?utf-8?Q?Campbell=2C_Lance?= Message-ID: In-Reply-To: References: Subject: Re: backup and restore with WAL files X-Mailer: Airmail (247) MIME-Version: 1.0 Content-Type: multipart/related; boundary="541c5193_6b94764_fd65" X-Pg-Spam-Score: -2.8 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org --541c5193_6b94764_fd65 Content-Type: multipart/alternative; boundary="541c5193_42c296bd_fd65" --541c5193_42c296bd_fd65 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline There can be some issues here:=C2=A0 Re: Storing the data. 1) great! i do the same.=C2=A0 2) Keep in mind that storing the wal files on the same server is not best p= ractice. What if the whole server blows up? Do you have a nas/shared drive = you can use?=C2=A0 3) This is ok. But=E2=80=A6=E2=80=A6 4) OK, now you can ONLY go back to the last base backup and forward from th= ere. The other 4 base backups are unusable. There are a few exceptions like= if you use the=C2=A0 =E2=80=9C-x=E2=80=9D option. read here:=C2=A0http://www.postgresql.org/docs= /9.2/static/app-pgbasebackup.html If you only want to go back to the last backup then its fine. But if you ne= ed to go back to a few days ago at a certain time you cannot, again the old= er 4 base backups are unusable without the wal files. To recover you need t= he "base backup + all the wal files=E2=80=9D one good read, there are many more:=C2=A0http://www.postgresql.org/docs/9.1= /static/continuous-archiving.html Your recovery procedure sounds good. =C2=A0Make sure all the file permissio= n are correct after the restore.=C2=A0 Let me know if you have any questions, hopefully I explained myself well -jason=C2=A0 On September 19, 2014 at 8:12:45 AM, Campbell, Lance (lance@illinois.edu) w= rote: PostgreSQL 9.3.5 RehHat =C2=A0 I am about to start using WAL archiving and pg_basebackup.=C2=A0 I have tes= ted the backing up of data.=C2=A0 I want to make sure I understand the prop= er way to restore.=C2=A0 As a note I use the database for web application h= osting.=C2=A0 The database runs on its own server.=C2=A0 =C2=A0 Storing data: 1)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I am going to do a nightly pg_basebackup. 2)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I am also shipping all WAL archives into t= heir own folder. 3)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I only keep the last five pg_basebackup . 4)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I delete all WAL archive files that have a= sequence ID less than the last pg_basebackup. =C2=A0 Recovery procedure if the worst happens: 1)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Stop the database. 2)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Rename the base data directory. 3)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Uncompress the last pg_basebackup into the= same directory as that of where my production data would normally be. 4)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 If the postgresql.conf file changes since = the last pg_basebackup I will then copy it over. 5)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I replace pg_hba.conf with pg_hba.conf.loc= al.=C2=A0 Only local users can access the database when started. 6)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I then start the database with the below r= ecovery.conf file: standby_mode =3D 'off' primary_conninfo =3D 'user=3Dxyz port=3D5432 sslmode=3Dprefer sslcompressio= n=3D1' restore_command =3D 'cp /xxx/yyy/wals/%f =C2=A0%p' 7)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I then make sure the database is working p= roperly locally. 8)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I then stop the database. 9)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Replace pg_hba.conf file with pg_hba.conf.= applications.=C2=A0 This file allows my application servers the ability to = connect to the database. 10)=C2=A0=C2=A0 Start the fully restored database. =C2=A0 I might need to add to the recovery.conf file a command to restore to a par= ticular date/time in order to not apply some dreadful situation that occurr= ed prior to the restore. =C2=A0 Did I miss anything?=C2=A0 I just wanted to make sure that all of the WAL f= iles get applied appropriately.=C2=A0 So if I have stored from the latest p= g_basebackup then all of the WAL files that have the same ID or greater wil= l be applied.=C2=A0 Not the WAL files that were older than the last pg_base= backup I am using.=C2=A0 Is that correct?=C2=A0 So if I actually restored f= rom a pg_basebackup that was a prior day then the WAL files that would be u= sed would not only be those related to that pg_basebackup but all of the fo= llowing days as well and the current ones.=C2=A0 Is that correct? =C2=A0 =C2=A0 Thanks, =C2=A0 Lance Campbell Software Architect Web Services at Public Affairs 217-333-0382 =20 =C2=A0 =C2=A0 --=20 =20 This transmission contains confidential and privileged information intended= =20 solely for the party identified above. If you receive this message in=20 error, you must not use it or convey it to others. Please destroy it=20 immediately and contact the sender at (303) 386-3955 or by return e-mail to= =20 the sender. --541c5193_42c296bd_fd65 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline =
There can be some issues here: 

Re: Storing the da= ta.
1) = great! i do the same. 
2) Keep in mind that storing the wal files on the sam= e server is not best practice. What if the whole server blows up? Do you ha= ve a nas/shared drive you can use? 
3) This is ok. But=E2=80=A6=E2=80=A6
4) OK, no= w you can ONLY go back to the last base backup and forward from there. The = other 4 base backups are unusable. There are a few exceptions like if you u= se the 
=E2=80=9C= -x=E2=80=9D option. read here: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html

If you only want to go = back to the last backup then its fine. But if you need to go back to a few = days ago at a certain time you cannot, again the older 4 base backups are u= nusable without the wal files. To recover you need the "base backup + all t= he wal files=E2=80=9D

=
Your re= covery procedure sounds good.  Make sure all the file permission are c= orrect after the restore. 

Let me know if you have any questions, hopefully I explained m= yself well
<= br>
-jason&n= bsp;

On September 19, 2014 at 8:12:45 AM, Camp= bell, Lance (lance@illinois.edu) = wrote:

PostgreSQL 9.3.5

RehHat

 

I am about to start using WAL archiving and pg_basebackup.  I have tested the backing up of data.  I want to make sure I understand the proper way to restore.  As a note I use the database for web application hosting.  The database runs on its own server. 

 

Storing data:

1)      I am going to do a nightly pg_basebackup.

2)      I am also shipping all WAL archives into their own folder.

3)      I only keep the last five pg_basebackup .

4)      I delete all WAL archive files that have a sequence ID less than the last pg_basebackup.

 

Recovery procedure if the worst happens:

1)      Stop the database.

2)      Rename the base data directory.

3)      Uncompress the last pg_basebackup into the same directory as that of where my production data would normally be.

4)      If the postgresql.conf file changes since the last pg_basebackup I will then copy it over.

5)      I replace pg_hba.conf with pg_hba.conf.local.  Only local users can access the database when started.

6)      I then start the database with the below recovery.conf file:

standby_mode =3D 'off'

primary_conninfo =3D 'user=3Dxyz port=3D5432 sslmode=3Dprefer sslcompression=3D1'

restore_command =3D 'cp /xxx/yyy/wals/%f  %p'

7)      I then make sure the database is working properly locally.

8)      I then stop the database.

9)      Replace pg_hba.conf file with pg_hba.conf.applications.  This file allows my application servers the ability to connect to the database.

10)   Start the fully restored database.

 

I might need to add to the recovery.conf file a command to restore to a particular date/time in order to not apply some dreadful situation that occurred prior to the restore.

 

Did I miss anything?  I just wanted to make sure that all of the WAL files get applied appropriately.  So if I have stored from the latest pg_basebackup then all of the WAL files that have the same ID or greater will be applied.  Not the WAL files that were older than the last pg_basebackup I am using.  Is that correct?  So if I actually restored from a pg_basebackup that was a prior day then the WAL files that would be used would not only be those related to that pg_basebackup but all of the following days as well and the current ones.  Is that correct?

 

 

Thanks,

 

Lance C= ampbell

Software Architect

Web Services at Public Affairs

217-333-0382

3D"University

 

 


This transmission contains confidential = and privileged information intended solely for the party identified above. If you receive = this message in error, you must not use it or convey it to others. Please destroy it immediately and contact the se= nder at (303) 386-3955 or by return e-mail to the sender.

--541c5193_42c296bd_fd65-- --541c5193_6b94764_fd65 Content-Type: application/octet-stream Content-Transfer-Encoding: base64 Content-ID: <0F752EA6-ED8A-462D-B35F-ACB84FA8E721> Content-Disposition: inline; filename="=?utf-8?Q?image001.png=4001CFD3E4.015A67E0?=" iVBORw0KGgoAAAANSUhEUgAAAJYAAAAZCAYAAADT59fvAAAFkElEQVRoge2aQWjbVhjH/xop85hD lMuQ2WEKdcG7rNKtYQmxTovZITIkdIVCLXoqZLi+ubQQB1aWm2Ma6KmVC4WsJBCH0dmnWiEt6Wl+ vQXqEfUUMQZVmQZqN3g7ODJ2otSSY3uk0w9MYiN9+kv663t63/eYhftP6GJRQy9YSMWRUySmJ8EC TjVDAPBV2MTlyB7e/HOmqyAjQ3/j4f5YT4UFnG6GAGDkzDv8/Mfn2H79WVdBJkd/x8iZdz0VFnC6 +ei/FhDwYTLk9mNc4JGeG8eLutH8bUrg27aR0mpfhQWcblyNtaBI4DkWpmWD51gADbMBgGnZKD3d HZjAgNOJq7G2iA5+WsBWbQ+vIqMYCYeaxpLSKkhLJgsIcMP1HSunVjF2MY9ihWBK4PHGsgEAGtED UwV4wtfLu26Y/dIR8IHhy1ivAmMFeCQoNwT0hcBYAX3BdVbYazIrFUpe7gMA2OEQ0rPjiAv8e3uK umHSxaIGff91c7/8fAI8x/rqRRbLNfqgQprfZyZiuD437itGt1pK27u0sL4DoFEHdOujKksl6sTl I6NQs7InbaZl0+W1HTyokGZJiNQN8ByL9OwFpBKi53PUiE4X1Wrzu3Augvz8tGcdytIG9H0TbDgE jehIJcTBGCs/P81IaZVqRMfeo4wnc/Acy6hZGeLVu5TUDbx+fANsOOS7wZ1KiAwb/oQmb63i+ty4 b1OdRIs8GWP4CEvFq3ehER1xcYwefqDUrMwkb61S538vekzLplJaBR9hUbt3ranFtGyaWalAWSph 68Ur6jVeXOAZKBKV0iquz417NhUAJG+ugh0OoXbvGgMApG5Q8erdwQ2FTuXeb8ZhwyHnb9erJtjh RoyRTz/uNkTXWoQoxzj7Jm+ugtQNenib82c5nD/LedYhpVWYlg01m2zTwoZDjJqVGXkyhmK5hpxa PXKsTvi5RrphUo3ouPKN2PxNiHKMmpX9GWvk4MIG+EfNJmFaNpQfN2Batu8b7lAs1yipG0jPXjjW 4Pn5BACgsP6828P4whnuHVIJkfFlLCHq/akKaEeejDFqVgapG07G6cpcm88a7TThXOTYbXiOZYQo B9OyoRG9axN3gudYJi7w0IgOKa1S3TCbxwqMNUBSCZFJJUSQugFlaaOrGOafjS5Ip3vBDmh02bh9 CUKUg0Z0jF3MN4ffY40lT8SQUyTohokvDmYdbDiEnCINRPCHipqVmVRCRGl7F8pSqets0qm1dng1 Sr9gwyGmdu8as5CKAwAWixoyKxXqaix5IoaZyS8xJfDgObY5nQUay4/3HmWC7HUC8vPTEKJcVy/Y zkTEKVEcxxbRG9sPKHPlFIk5mKFieW3HPWNNiWNtZgIaDWjnE/QMTwYbDjHVggIhymGxqGHTxzKk ma9jAIDW2pwbTk3LmZX2g9L2bttDIUS5ZuZyrWNl7pR7LuLNX297Gk8jOi2s72Djh0snunBSWqX5 7xN9vQFAY2reWmphwyFGvZGkzjKkmYmYpziphMgU1p9TjegolmvUrRCaU6vUtGxs3L7UwzM4SmF9 B+xwqK02x3OjAAbU0jEtm2q1PQBwreO8bz8nOy6v7VCN6M1P5k4ZU+d5T3Gcqv/m0120xlhe26Gk bngyVauW1tlPx2MfnK9z/q0IUY6pFhTfw5WT7ZSlkmMi6mjMqVVaWH8ONSt37G604gytL37zvixq SuDbanOmZdPFYhXyZAzMwv0ntPjTY1yOHD1xPzzcH0Pqu29d2xZSWm27EVemhY4tB1I3aKfMqd5I diy4traT3Diu1dJJi5csd3i/41olxXKNmpbtuytQLNfo5rNdkJcGTMuGEOUgnIsgPXvBVyH6cNsL AKoFxdPDllmpgLzch26Y4DnWaZmBUX/59UjQbvFimID/B/8CO2CIAn28iA0AAAAASUVORK5CYII= --541c5193_6b94764_fd65--