From d.wall@computer.org Fri Jan 11 20:28:19 2008 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id F1F6B2E49DD for ; Fri, 11 Jan 2008 16:28:19 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 78882-10 for ; Fri, 11 Jan 2008 16:28:15 -0400 (AST) Received: from mail2.yozons.com (mail2.yozons.com [63.251.168.97]) by postgresql.org (Postfix) with ESMTP id AF98E2E453C for ; Fri, 11 Jan 2008 16:28:15 -0400 (AST) Received: from [192.168.1.2] (pool-71-164-18-27.sttlwa.fios.verizon.net [71.164.18.27]) (authenticated bits=0) by mail2.yozons.com (8.12.10/8.12.10) with ESMTP id m0BKSElG026702 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Fri, 11 Jan 2008 12:28:14 -0800 Message-ID: <4787D161.9000000@computer.org> Date: Fri, 11 Jan 2008 12:28:17 -0800 From: David Wall Reply-To: d.wall@computer.org User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: pgsql-admin@postgresql.org Subject: Re: WAL recovery, stop and resume recovery? References: <4787C7A7.3000503@computer.org> In-Reply-To: <4787C7A7.3000503@computer.org> Content-Type: multipart/alternative; boundary="------------060305090000020206030902" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200801/124 X-Sequence-Number: 27898 This is a multi-part message in MIME format. --------------060305090000020206030902 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit > LOG: database system was shut down at 2008-01-11 11:40:05 PST > LOG: starting archive recovery > LOG: restore_command = "~/postgresql/bin/pg_standby -l -d -s 2 -k 20 > -t ~/postgresql/restoreWALs/STOP_RESTORE ~/postgresql/restoreWALs %f > %p 2>> ~/pg_standby.log" > *LOG: restored log file "00000001000000050000001D" from archive > LOG: invalid record length at 5/1D000068 > LOG: invalid primary checkpoint record > LOG: restored log file "00000001000000050000001D" from archive > LOG: invalid resource manager ID in secondary checkpoint record > PANIC: could not locate a valid checkpoint record* > LOG: startup process (PID 9219) was terminated by signal 6 > LOG: aborting startup due to startup process failure > LOG: logger shutting down One more thing is that the error above about the 1D log file above having an invalid record length, etc. is interesting in that if I restore from the TAR backup created for this and start the backup database in recover mode, it manages to process the 1D log file just fine (along with all the previous log files of course). David --------------060305090000020206030902 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
LOG:  database system was shut down at 2008-01-11 11:40:05 PST
LOG:  starting archive recovery
LOG:  restore_command = "~/postgresql/bin/pg_standby -l -d -s 2 -k 20 -t ~/postgresql/restoreWALs/STOP_RESTORE ~/postgresql/restoreWALs %f %p 2>> ~/pg_standby.log"
LOG:  restored log file "00000001000000050000001D" from archive
LOG:  invalid record length at 5/1D000068
LOG:  invalid primary checkpoint record
LOG:  restored log file "00000001000000050000001D" from archive
LOG:  invalid resource manager ID in secondary checkpoint record
PANIC:  could not locate a valid checkpoint record

LOG:  startup process (PID 9219) was terminated by signal 6
LOG:  aborting startup due to startup process failure
LOG:  logger shutting down

One more thing is that the error above about the 1D log file above having an invalid record length, etc. is interesting in that if I restore from the TAR backup created for this and start the backup database in recover mode, it manages to process the 1D log file just fine (along with all the previous log files of course).

David
--------------060305090000020206030902-- From tgl@sss.pgh.pa.us Fri Jan 11 20:34:04 2008 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 7ADDF2E0264 for ; Fri, 11 Jan 2008 16:34:04 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 81361-02 for ; Fri, 11 Jan 2008 16:34:01 -0400 (AST) Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id D0B3B2E01B5 for ; Fri, 11 Jan 2008 16:34:01 -0400 (AST) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.1/8.14.1) with ESMTP id m0BKXpdg024389; Fri, 11 Jan 2008 15:33:51 -0500 (EST) To: d.wall@computer.org cc: pgsql-admin@postgresql.org Subject: Re: WAL recovery, stop and resume recovery? In-reply-to: <4787C7A7.3000503@computer.org> References: <4787C7A7.3000503@computer.org> Comments: In-reply-to David Wall message dated "Fri, 11 Jan 2008 11:46:47 -0800" Date: Fri, 11 Jan 2008 15:33:51 -0500 Message-ID: <24388.1200083631@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200801/125 X-Sequence-Number: 27899 David Wall writes: > Is it allowable to have a backup database in recovery mode, then stop > recovery (in this case, by putting the trigger file in place to stop > pg_standby), check out that the backup db appears up to date, stop the > now active backup db, and then restart it in recover mode again to have > it resume its backup role? No. Once you've done any transactions in the backup DB, its transaction history has diverged from the master and you can't resume tracking the master. It shouldn't even let you try --- what shenanigans did you pull to force it back into recovery mode? There's some work being done on allowing read-only queries against an in-recovery database, which I think would satisfy your desire to see if the backup were sane or not. But I wouldn't bet money on that getting into the system anytime soon. It's definitely not something you can cobble up from spare parts. regards, tom lane From d.wall@computer.org Fri Jan 11 21:08:06 2008 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 166932E49D9 for ; Fri, 11 Jan 2008 17:08:06 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 85997-01 for ; Fri, 11 Jan 2008 17:08:00 -0400 (AST) Received: from mail2.yozons.com (mail2.yozons.com [63.251.168.97]) by postgresql.org (Postfix) with ESMTP id C09802E49C3 for ; Fri, 11 Jan 2008 17:08:02 -0400 (AST) Received: from [192.168.1.2] (pool-71-164-18-27.sttlwa.fios.verizon.net [71.164.18.27]) (authenticated bits=0) by mail2.yozons.com (8.12.10/8.12.10) with ESMTP id m0BL81lG026796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Fri, 11 Jan 2008 13:08:01 -0800 Message-ID: <4787DAB4.9020402@computer.org> Date: Fri, 11 Jan 2008 13:08:04 -0800 From: David Wall Reply-To: d.wall@computer.org User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: pgsql-admin@postgresql.org Subject: Re: WAL recovery, stop and resume recovery? References: <4787C7A7.3000503@computer.org> <24388.1200083631@sss.pgh.pa.us> In-Reply-To: <24388.1200083631@sss.pgh.pa.us> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200801/126 X-Sequence-Number: 27900 > No. Once you've done any transactions in the backup DB, its transaction > history has diverged from the master and you can't resume tracking the > master. It shouldn't even let you try --- what shenanigans did you pull > to force it back into recovery mode? > Well, I didn't think it was shenanigans, I just stopped the database once it completed the first recovery, ran a few queries, then re-installed the recovery.conf and started it back up like I initially did. I figured this could be an issue, but since I hadn't issued any changes, I had hoped it might work. > There's some work being done on allowing read-only queries against an > in-recovery database, which I think would satisfy your desire to see if > the backup were sane or not. But I wouldn't bet money on that getting > into the system anytime soon. It's definitely not something you can > cobble up from spare parts. > Fair enough. It's probably not a big deal as I'm doing this only because we're new to using WAL copying for a warm standby, and of course we're testing to see that rows inserted, removed, updated, tables added and dropped, indexes added and dropped, etc. are all making it through. It appears that this works like a charm! Is there a way to know how many WAL files I should keep around to ensure I can recover back to a valid primary checkpoint without having to redo the entire backup process on the primary in 8.2, or do I just have to wait for 8.3 and %r option for recovery? Thanks, David