Received: from localhost (maia-3.hub.org [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 5E0E89FB1BF for ; Mon, 8 Jan 2007 12:41:54 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-new, port 10024) with ESMTP id 50933-10 for ; Mon, 8 Jan 2007 12:41:32 -0400 (AST) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.4 Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.191]) by postgresql.org (Postfix) with ESMTP id 534459FA4C1 for ; Mon, 8 Jan 2007 12:41:32 -0400 (AST) Received: by nf-out-0910.google.com with SMTP id x4so9554987nfb for ; Mon, 08 Jan 2007 08:41:31 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type; b=bVH2ceiv0sKP3G+po2Dya9KDr8aAf+sZXDNqvxcV6Nnt8oVZZAck/VzpfYUMGOJDSaNCO1Pmw2NaiWt0/jnel7BrS2QPQ1OScx7weX9MbcLVto3IdvCw9j6m2BSJR/13uAkXWPgZNmveoPPMJ32ND6kyhHdBXrVyuy92gBzL2Bs= Received: by 10.82.118.2 with SMTP id q2mr2712437buc.1168274490981; Mon, 08 Jan 2007 08:41:30 -0800 (PST) Received: by 10.82.112.17 with HTTP; Mon, 8 Jan 2007 08:41:30 -0800 (PST) Message-ID: <1d219a6f0701080841l3a795a08se3852c23bea2e240@mail.gmail.com> Date: Mon, 8 Jan 2007 11:41:30 -0500 From: "Chris Hoover" To: "pgsql-admin@postgresql.org" Subject: 8.1.3 Crash/Corruption Problem MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_59758_22071172.1168274490614" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.256 tagged_above=0 required=5 tests=AWL, BAYES_20, HTML_10_20, HTML_MESSAGE, NORMAL_HTTP_TO_IP, SPF_PASS X-Spam-Level: X-Archive-Number: 200701/64 X-Sequence-Number: 24220 ------=_Part_59758_22071172.1168274490614 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Over the weekend, I suffered from major database corruption on my 8.1.3database. Fortunetly, this was my development/non-production box so I did not have to spend my entire weekend getting it back online, but I did spend a major portion of it. I ended up having to do a pitr recovery of the system. I don't understand what happened to the system. Please help explain what happened. Also, would there have been another option to get the system back online w/o the full recovery? Thanks, Chris Here is the log of the crash: ,19684,,2007-01-06 03:20:04.542 EST,,70594,,,,DEBUG: server process (PID 9375) exited with exit code 0 ,19684,,2007-01-06 03:20:04.547 EST,,70595,,,,DEBUG: forked new backend, pid=9380 socket=8 [unknown],9380,[unknown],2007-01-06 03:20:04.547 EST,459f5bb4.24a4,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9380,postgres,2007-01-06 03:20:04.547 EST,459f5bb4.24a4,2,2007-01-06 03:20:04 EST,0,authentication,LOG: connection authorized: user=postgres database=dba_stats dba_stats,9380,postgres,2007-01-06 03:20:04.549 EST,459f5bb4.24a4,3,2007-01-06 03:20:04 EST,322692053,idle,LOG: statement: insert into db_transactions values ('192.168.1.73','d45','0','79443','5','1769025','14411089','01/06/2007 03:20:01'); dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,4,2007-01-06 03:20:04 EST,0,INSERT,LOG: duration: 1.974 ms dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,5,2007-01-06 03:20:04 EST,0,INSERT,LOG: duration: 1.974 ms statement: insert into db_transactions values ('192.168.1.73','d45','0','79443','5','1769025','14411089','01/06/2007 03:20:01'); dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,6,2007-01-06 03:20:04 EST,0,idle,LOG: disconnection: session time: 0:00:00.00user=postgres database=dba_stats host=[local] ,19684,,2007-01-06 03:20:04.552 EST,,70596,,,,DEBUG: server process (PID 9380) exited with exit code 0 [unknown],9385,[unknown],2007-01-06 03:20:04.557 EST,459f5bb4.24a9,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9385,postgres,2007-01-06 03:20:04.557 EST,459f5bb4.24a9,2,2007-01-06 03:20:04 EST,0,authentication,LOG: connection authorized: user=postgres database=dba_stats ,19684,,2007-01-06 03:20:04.558 EST,,70597,,,,DEBUG: forked new backend, pid=9385 socket=8 dba_stats,9385,postgres,2007-01-06 03:20:04.560 EST,459f5bb4.24a9,3,2007-01-06 03:20:04 EST,322692055,idle,LOG: statement: insert into db_transactions values ('192.168.1.73','d46','0','116796','112','7246902','23447082','01/06/2007 03:20:01'); dba_stats,9385,postgres,2007-01-06 03:20:04.561 EST,459f5bb4.24a9,4,2007-01-06 03:20:04 EST,322692055,INSERT,PANIC: right sibling's left-link doesn't match dba_stats,9385,postgres,2007-01-06 03:20:04.561 EST,459f5bb4.24a9,5,2007-01-06 03:20:04 EST,322692055,INSERT,STATEMENT: insert into db_transactions values ('192.168.1.73','d46','0','116796','112','7246902','23447082','01/06/2007 03:20:01'); ,19684,,2007-01-06 03:20:04.562 EST,,70598,,,,DEBUG: server process (PID 9385) was terminated by signal 6 ,19684,,2007-01-06 03:20:04.562 EST,,70599,,,,LOG: server process (PID 9385) was terminated by signal 6 ,19684,,2007-01-06 03:20:04.562 EST,,70600,,,,LOG: terminating any other active server processes ,19684,,2007-01-06 03:20:04.562 EST,,70601,,,,DEBUG: sending SIGQUIT to process 8277 ,19684,,2007-01-06 03:20:04.562 EST,,70602,,,,DEBUG: sending SIGQUIT to process 8008 ,19684,,2007-01-06 03:20:04.562 EST,,70603,,,,DEBUG: sending SIGQUIT to process 8227 ,19684,,2007-01-06 03:20:04.562 EST,,70604,,,,DEBUG: sending SIGQUIT to process 8275 sys_logon,8277,postgres,2007-01-06 03:20:04.562 EST,459f5ad5.2055,13,2007-01-06 03:16:21 EST,0,idle,WARNING: terminating connection because of crash of another server process sys_logon,8277,postgres,2007-01-06 03:20:04.562 EST,459f5ad5.2055,14,2007-01-06 03:16:21 EST,0,idle,DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. sys_logon,8277,postgres,2007-01-06 03:20:04.562 EST,459f5ad5.2055,15,2007-01-06 03:16:21 EST,0,idle,HINT: In a moment you should be able to reconnect to the database and repeat your command. ,19684,,2007-01-06 03:20:04.564 EST,,70605,,,,DEBUG: server process (PID 8277) exited with exit code 1 ,19684,,2007-01-06 03:20:04.564 EST,,70606,,,,LOG: all server processes terminated; reinitializing ,9390,,2007-01-06 03:20:04.603 EST,,1,,,,LOG: database system was interrupted at 2007-01-06 03:15:04 EST ,9390,,2007-01-06 03:20:04.603 EST,,2,,,,LOG: checkpoint record is at 299/3856B2DC ,9390,,2007-01-06 03:20:04.603 EST,,3,,,,LOG: redo record is at 299/3856B2DC; undo record is at 0/0; shutdown TRUE ,9390,,2007-01-06 03:20:04.603 EST,,4,,,,LOG: next transaction ID: 322691436; next OID: 79454488 ,9390,,2007-01-06 03:20:04.603 EST,,5,,,,LOG: next MultiXactId: 9; next MultiXactOffset: 17 ,9390,,2007-01-06 03:20:04.603 EST,,6,,,,LOG: database system was not properly shut down; automatic recovery in progress [unknown],9391,[unknown],2007-01-06 03:20:04.604 EST,459f5bb4.24af,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9391,postgres,2007-01-06 03:20:04.604 EST,459f5bb4.24af,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.604 EST,,70607,,,,DEBUG: forked new backend, pid=9391 socket=8 ,19684,,2007-01-06 03:20:04.605 EST,,70608,,,,DEBUG: server process (PID 9391) exited with exit code 0 ,9390,,2007-01-06 03:20:04.607 EST,,7,,,,LOG: redo starts at 299/3856B320 [unknown],9396,[unknown],2007-01-06 03:20:04.612 EST,459f5bb4.24b4,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9396,postgres,2007-01-06 03:20:04.612 EST,459f5bb4.24b4,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.614 EST,,70609,,,,DEBUG: forked new backend, pid=9396 socket=8 ,19684,,2007-01-06 03:20:04.614 EST,,70610,,,,DEBUG: server process (PID 9396) exited with exit code 0 [unknown],9401,[unknown],2007-01-06 03:20:04.620 EST,459f5bb4.24b9,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9401,postgres,2007-01-06 03:20:04.620 EST,459f5bb4.24b9,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.622 EST,,70611,,,,DEBUG: forked new backend, pid=9401 socket=8 ,19684,,2007-01-06 03:20:04.622 EST,,70612,,,,DEBUG: server process (PID 9401) exited with exit code 0 ,9390,,2007-01-06 03:20:04.622 EST,,8,,,,LOG: record with zero length at 299/386E998C ,9390,,2007-01-06 03:20:04.622 EST,,9,,,,LOG: redo done at 299/386E9964 [unknown],9406,[unknown],2007-01-06 03:20:04.628 EST,459f5bb4.24be,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9406,postgres,2007-01-06 03:20:04.629 EST,459f5bb4.24be,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.630 EST,,70613,,,,DEBUG: forked new backend, pid=9406 socket=8 ,19684,,2007-01-06 03:20:04.630 EST,,70614,,,,DEBUG: server process (PID 9406) exited with exit code 0 [unknown],9411,[unknown],2007-01-06 03:20:04.636 EST,459f5bb4.24c3,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9411,postgres,2007-01-06 03:20:04.636 EST,459f5bb4.24c3,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.638 EST,,70615,,,,DEBUG: forked new backend, pid=9411 socket=8 ,19684,,2007-01-06 03:20:04.638 EST,,70616,,,,DEBUG: server process (PID 9411) exited with exit code 0 [unknown],9416,[unknown],2007-01-06 03:20:04.643 EST,459f5bb4.24c8,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9416,postgres,2007-01-06 03:20:04.644 EST,459f5bb4.24c8,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.645 EST,,70617,,,,DEBUG: forked new backend, pid=9416 socket=8 ,19684,,2007-01-06 03:20:04.645 EST,,70618,,,,DEBUG: server process (PID 9416) exited with exit code 0 [unknown],9421,[unknown],2007-01-06 03:20:04.652 EST,459f5bb4.24cd,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9421,postgres,2007-01-06 03:20:04.652 EST,459f5bb4.24cd,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.655 EST,,70619,,,,DEBUG: forked new backend, pid=9421 socket=8 ,19684,,2007-01-06 03:20:04.655 EST,,70620,,,,DEBUG: server process (PID 9421) exited with exit code 0 [unknown],9426,[unknown],2007-01-06 03:20:04.660 EST,459f5bb4.24d2,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9426,postgres,2007-01-06 03:20:04.661 EST,459f5bb4.24d2,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,19684,,2007-01-06 03:20:04.662 EST,,70621,,,,DEBUG: forked new backend, pid=9426 socket=8 ,19684,,2007-01-06 03:20:04.662 EST,,70622,,,,DEBUG: server process (PID 9426) exited with exit code 0 [unknown],9431,[unknown],2007-01-06 03:20:04.668 EST,459f5bb4.24d7,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG: connection received: host=[local] dba_stats,9431,postgres,2007-01-06 03:20:04.668 EST,459f5bb4.24d7,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL: the database system is starting up ,9390,,2007-01-06 03:20:04.669 EST,,10,,,,PANIC: failed to re-find parent key in "3000108689" ,19684,,2007-01-06 03:20:04.669 EST,,70623,,,,DEBUG: forked new backend, pid=9431 socket=8 ,19684,,2007-01-06 03:20:04.669 EST,,70624,,,,DEBUG: server process (PID 9431) exited with exit code 0 ,19684,,2007-01-06 03:20:04.673 EST,,70625,,,,LOG: startup process (PID 9390) was terminated by signal 6 ,19684,,2007-01-06 03:20:04.673 EST,,70626,,,,LOG: aborting startup due to startup process failure ,19685,,2007-01-06 03:20:04.681 EST,,1,,,,LOG: logger shutting down Here is the log of the first try at starting the database: ,2960,,2007-01-06 08:59:13.996 EST,,1,,,,LOG: database system was interrupted while in recovery at 2007-01-06 03:20:04 EST ,2960,,2007-01-06 08:59:13.996 EST,,2,,,,HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery. ,2960,,2007-01-06 08:59:13.996 EST,,3,,,,LOG: checkpoint record is at 299/3856B2DC ,2960,,2007-01-06 08:59:13.996 EST,,4,,,,LOG: redo record is at 299/3856B2DC; undo record is at 0/0; shutdown TRUE ,2960,,2007-01-06 08:59:13.996 EST,,5,,,,LOG: next transaction ID: 322691436; next OID: 79454488 ,2960,,2007-01-06 08:59:13.996 EST,,6,,,,LOG: next MultiXactId: 9; next MultiXactOffset: 17 ,2960,,2007-01-06 08:59:13.996 EST,,7,,,,LOG: database system was not properly shut down; automatic recovery in progress ,2960,,2007-01-06 08:59:14.192 EST,,8,,,,LOG: redo starts at 299/3856B320 ,2960,,2007-01-06 08:59:14.205 EST,,9,,,,LOG: record with zero length at 299/386E998C ,2960,,2007-01-06 08:59:14.205 EST,,10,,,,LOG: redo done at 299/386E9964 ,2960,,2007-01-06 08:59:14.243 EST,,11,,,,PANIC: failed to re-find parent key in "3000108689" ,2956,,2007-01-06 08:59:14.246 EST,,2,,,,LOG: startup process (PID 2960) was terminated by signal 6 ,2956,,2007-01-06 08:59:14.246 EST,,3,,,,LOG: aborting startup due to startup process failure ,2959,,2007-01-06 08:59:14.254 EST,,1,,,,LOG: logger shutting down ------=_Part_59758_22071172.1168274490614 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Over the weekend, I suffered from major database corruption on my 8.1.3 database.  Fortunetly, this was my development/non-production box so I did not have to spend my entire weekend getting it back online, but I did spend a major portion of it.  I ended up having to do a pitr recovery of the system.  I don't understand what happened to the system.  Please help explain what happened.  Also, would there have been another option to get the system back online w/o the full recovery?

Thanks,

Chris


Here is the log of the crash:

,19684,,2007-01-06 03:20:04.542 EST,,70594,,,,DEBUG:  server process (PID 9375) exited with exit code 0
,19684,,2007-01-06 03:20:04.547 EST,,70595,,,,DEBUG:  forked new backend, pid=9380 socket=8
[unknown],9380,[unknown],2007-01-06 03:20:04.547 EST,459f5bb4.24a4,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9380,postgres,2007-01-06 03:20:04.547 EST,459f5bb4.24a4 ,2,2007-01-06 03:20:04 EST,0,authentication,LOG:  connection authorized: user=postgres database=dba_stats
dba_stats,9380,postgres,2007-01-06 03:20:04.549 EST,459f5bb4.24a4,3,2007-01-06 03:20:04 EST,322692053,idle,LOG:  statement: insert into db_transactions values (' 192.168.1.73','d45','0','79443','5','1769025','14411089','01/06/2007 03:20:01');
dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,4,2007-01-06 03:20:04 EST,0,INSERT,LOG:  duration: 1.974 ms
dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,5,2007-01-06 03:20:04 EST,0,INSERT,LOG:  duration: 1.974 ms  statement: insert into db_transactions values (' 192.168.1.73','d45','0','79443','5','1769025','14411089','01/06/2007 03:20:01');
dba_stats,9380,postgres,2007-01-06 03:20:04.551 EST,459f5bb4.24a4,6,2007-01-06 03:20:04 EST,0,idle,LOG:  disconnection: session time: 0:00:00.00 user=postgres database=dba_stats host=[local]
,19684,,2007-01-06 03:20:04.552 EST,,70596,,,,DEBUG:  server process (PID 9380) exited with exit code 0
[unknown],9385,[unknown],2007-01-06 03:20:04.557 EST,459f5bb4.24a9,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9385,postgres,2007-01-06 03:20:04.557 EST,459f5bb4.24a9 ,2,2007-01-06 03:20:04 EST,0,authentication,LOG:  connection authorized: user=postgres database=dba_stats
,19684,,2007-01-06 03:20:04.558 EST,,70597,,,,DEBUG:  forked new backend, pid=9385 socket=8
dba_stats,9385,postgres,2007-01-06 03:20: 04.560 EST,459f5bb4.24a9,3,2007-01-06 03:20:04 EST,322692055,idle,LOG:  statement: insert into db_transactions values ('192.168.1.73','d46','0','116796','112','7246902','23447082','01/06/2007 03:20:01');
dba_stats,9385,postgres,2007-01-06 03:20:04.561 EST,459f5bb4.24a9,4,2007-01-06 03:20:04 EST,322692055,INSERT,PANIC:  right sibling's left-link doesn't match
dba_stats,9385,postgres,2007-01-06 03:20:04.561 EST, 459f5bb4.24a9,5,2007-01-06 03:20:04 EST,322692055,INSERT,STATEMENT:  insert into db_transactions values ('192.168.1.73','d46','0','116796','112','7246902','23447082','01/06/2007 03:20:01');
,19684,,2007-01-06 03:20:04.562 EST,,70598,,,,DEBUG:  server process (PID 9385) was terminated by signal 6
,19684,,2007-01-06 03:20:04.562 EST,,70599,,,,LOG:  server process (PID 9385) was terminated by signal 6
,19684,,2007-01-06 03:20:04.562 EST,,70600,,,,LOG:  terminating any other active server processes
,19684,,2007-01-06 03:20:04.562 EST,,70601,,,,DEBUG:  sending SIGQUIT to process 8277
,19684,,2007-01-06 03:20:04.562 EST,,70602,,,,DEBUG:  sending SIGQUIT to process 8008
,19684,,2007-01-06 03:20:04.562 EST,,70603,,,,DEBUG:  sending SIGQUIT to process 8227
,19684,,2007-01-06 03:20:04.562 EST,,70604,,,,DEBUG:  sending SIGQUIT to process 8275
sys_logon,8277,postgres,2007-01-06 03:20:04.562 EST,459f5ad5.2055,13,2007-01-06 03:16:21 EST,0,idle,WARNING:  terminating connection because of crash of another server process
sys_logon,8277,postgres,2007-01-06 03:20: 04.562 EST,459f5ad5.2055,14,2007-01-06 03:16:21 EST,0,idle,DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
sys_logon,8277,postgres,2007-01-06 03:20:04.562 EST,459f5ad5.2055,15,2007-01-06 03:16:21 EST,0,idle,HINT:  In a moment you should be able to reconnect to the database and repeat your command.
,19684,,2007-01-06 03:20: 04.564 EST,,70605,,,,DEBUG:  server process (PID 8277) exited with exit code 1
,19684,,2007-01-06 03:20:04.564 EST,,70606,,,,LOG:  all server processes terminated; reinitializing
,9390,,2007-01-06 03:20:04.603 EST,,1,,,,LOG:  database system was interrupted at 2007-01-06 03:15:04 EST
,9390,,2007-01-06 03:20:04.603 EST,,2,,,,LOG:  checkpoint record is at 299/3856B2DC
,9390,,2007-01-06 03:20:04.603 EST,,3,,,,LOG:  redo record is at 299/3856B2DC; undo record is at 0/0; shutdown TRUE
,9390,,2007-01-06 03:20: 04.603 EST,,4,,,,LOG:  next transaction ID: 322691436; next OID: 79454488
,9390,,2007-01-06 03:20:04.603 EST,,5,,,,LOG:  next MultiXactId: 9; next MultiXactOffset: 17
,9390,,2007-01-06 03:20:04.603 EST,,6,,,,LOG:  database system was not properly shut down; automatic recovery in progress
[unknown],9391,[unknown],2007-01-06 03:20:04.604 EST,459f5bb4.24af,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9391,postgres,2007-01-06 03:20:04.604 EST,459f5bb4.24af ,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.604 EST,,70607,,,,DEBUG:  forked new backend, pid=9391 socket=8
,19684,,2007-01-06 03:20:04.605 EST,,70608,,,,DEBUG:  server process (PID 9391) exited with exit code 0
,9390,,2007-01-06 03:20:04.607 EST,,7,,,,LOG:  redo starts at 299/3856B320
[unknown],9396,[unknown],2007-01-06 03:20:04.612 EST,459f5bb4.24b4,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9396,postgres,2007-01-06 03:20:04.612 EST,459f5bb4.24b4,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.614 EST,,70609,,,,DEBUG:  forked new backend, pid=9396 socket=8
,19684,,2007-01-06 03:20:04.614 EST,,70610,,,,DEBUG:  server process (PID 9396) exited with exit code 0
[unknown],9401,[unknown],2007-01-06 03:20:04.620 EST,459f5bb4.24b9,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9401,postgres,2007-01-06 03:20:04.620 EST,459f5bb4.24b9,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.622 EST,,70611,,,,DEBUG:  forked new backend, pid=9401 socket=8
,19684,,2007-01-06 03:20:04.622 EST,,70612,,,,DEBUG:  server process (PID 9401) exited with exit code 0
,9390,,2007-01-06 03:20:04.622 EST,,8,,,,LOG:  record with zero length at 299/386E998C
,9390,,2007-01-06 03:20: 04.622 EST,,9,,,,LOG:  redo done at 299/386E9964
[unknown],9406,[unknown],2007-01-06 03:20:04.628 EST,459f5bb4.24be,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9406,postgres,2007-01-06 03:20: 04.629 EST,459f5bb4.24be,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.630 EST,,70613,,,,DEBUG:  forked new backend, pid=9406 socket=8
,19684,,2007-01-06 03:20: 04.630 EST,,70614,,,,DEBUG:  server process (PID 9406) exited with exit code 0
[unknown],9411,[unknown],2007-01-06 03:20:04.636 EST,459f5bb4.24c3,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9411,postgres,2007-01-06 03:20:04.636 EST,459f5bb4.24c3,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.638 EST,,70615,,,,DEBUG:  forked new backend, pid=9411 socket=8
,19684,,2007-01-06 03:20:04.638 EST,,70616,,,,DEBUG:  server process (PID 9411) exited with exit code 0
[unknown],9416,[unknown],2007-01-06 03:20:04.643 EST,459f5bb4.24c8,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9416,postgres,2007-01-06 03:20:04.644 EST,459f5bb4.24c8,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.645 EST,,70617,,,,DEBUG:  forked new backend, pid=9416 socket=8
,19684,,2007-01-06 03:20:04.645 EST,,70618,,,,DEBUG:  server process (PID 9416) exited with exit code 0
[unknown],9421,[unknown],2007-01-06 03:20:04.652 EST,459f5bb4.24cd,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9421,postgres,2007-01-06 03:20:04.652 EST,459f5bb4.24cd,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.655 EST,,70619,,,,DEBUG:  forked new backend, pid=9421 socket=8
,19684,,2007-01-06 03:20:04.655 EST,,70620,,,,DEBUG:  server process (PID 9421) exited with exit code 0
[unknown],9426,[unknown],2007-01-06 03:20:04.660 EST,459f5bb4.24d2,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9426,postgres,2007-01-06 03:20:04.661 EST,459f5bb4.24d2,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,19684,,2007-01-06 03:20:04.662 EST,,70621,,,,DEBUG:  forked new backend, pid=9426 socket=8
,19684,,2007-01-06 03:20:04.662 EST,,70622,,,,DEBUG:  server process (PID 9426) exited with exit code 0
[unknown],9431,[unknown],2007-01-06 03:20:04.668 EST,459f5bb4.24d7,1,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,LOG:  connection received: host=[local]
dba_stats,9431,postgres,2007-01-06 03:20:04.668 EST,459f5bb4.24d7,2,2007-01-06 03:20:04 EST,0,/usr/bin/postmaster,FATAL:  the database system is starting up
,9390,,2007-01-06 03:20:04.669 EST,,10,,,,PANIC:  failed to re-find parent key in "3000108689"
,19684,,2007-01-06 03:20:04.669 EST,,70623,,,,DEBUG:  forked new backend, pid=9431 socket=8
,19684,,2007-01-06 03:20:04.669 EST,,70624,,,,DEBUG:  server process (PID 9431) exited with exit code 0
,19684,,2007-01-06 03:20: 04.673 EST,,70625,,,,LOG:  startup process (PID 9390) was terminated by signal 6
,19684,,2007-01-06 03:20:04.673 EST,,70626,,,,LOG:  aborting startup due to startup process failure
,19685,,2007-01-06 03:20:04.681 EST,,1,,,,LOG:  logger shutting down


Here is the log of the first try at starting the database:

,2960,,2007-01-06 08:59:13.996 EST,,1,,,,LOG:  database system was interrupted while in recovery at 2007-01-06 03:20:04 EST
,2960,,2007-01-06 08:59: 13.996 EST,,2,,,,HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
,2960,,2007-01-06 08:59:13.996 EST,,3,,,,LOG:  checkpoint record is at 299/3856B2DC
,2960,,2007-01-06 08:59: 13.996 EST,,4,,,,LOG:  redo record is at 299/3856B2DC; undo record is at 0/0; shutdown TRUE
,2960,,2007-01-06 08:59:13.996 EST,,5,,,,LOG:  next transaction ID: 322691436; next OID: 79454488
,2960,,2007-01-06 08:59:13.996 EST,,6,,,,LOG:  next MultiXactId: 9; next MultiXactOffset: 17
,2960,,2007-01-06 08:59:13.996 EST,,7,,,,LOG:  database system was not properly shut down; automatic recovery in progress
,2960,,2007-01-06 08:59:14.192 EST,,8,,,,LOG:  redo starts at 299/3856B320
,2960,,2007-01-06 08:59:14.205 EST,,9,,,,LOG:  record with zero length at 299/386E998C
,2960,,2007-01-06 08:59:14.205 EST,,10,,,,LOG:  redo done at 299/386E9964
,2960,,2007-01-06 08:59: 14.243 EST,,11,,,,PANIC:  failed to re-find parent key in "3000108689"
,2956,,2007-01-06 08:59:14.246 EST,,2,,,,LOG:  startup process (PID 2960) was terminated by signal 6
,2956,,2007-01-06 08:59:14.246 EST,,3,,,,LOG:  aborting startup due to startup process failure
,2959,,2007-01-06 08:59:14.254 EST,,1,,,,LOG:  logger shutting down

------=_Part_59758_22071172.1168274490614--