pg.ddx.io pgsql-admin@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Help desperately needed: reoccurring invalid page header 3+ messages / 2 participants [nested] [flat]
* Help desperately needed: reoccurring invalid page header @ 2005-04-28 06:27 Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> 2005-04-28 12:58 ` Re: Help desperately needed: reoccurring invalid page Ian Westmacott <ianw@intellivid.com> 0 siblings, 1 reply; 3+ messages in thread From: Mauri Sahlberg @ 2005-04-28 06:27 UTC (permalink / raw) To: pgsql-admin; +Cc: peikko@claymountain.com, PoRRo <theporro@saunalahti.fi> Hi, Problem environment: Database-server: - Fedora Core 1 under virtuozzo virtual server running on Fujitsu-siemens hardware - Kernel: Linux claymountain.planeetta.com 2.4.20-021stab028.5.777-enterprise #1 SMP Tue Feb 22 17:44:46 MSK 2005 i686 i686 i386 GNU/Linux - Postgresql 7.4.6 (from rpm by provider: release: 2PGDG Build Date: Mon 25 Oct 2004 11:25:56 PM EEST Build host gunduz.org) - Query and update clients as php-applications Database-update-client: - Fedora Core 2 - Kernel Linux error.may.day 2.6.10-1.771_FC2 #1 Mon Mar 28 00:50:14 EST 2005 i686 athlon i386 GNU/Linux - A c++-client build against libpq++-4.0 from pgfoundry. I can not remember when the application was compiled but definitely not after auto upgrading client version to 7.4.7 - psql (PostgreSQL) 7.4.7 Problem description: - After running out of money and several years of operation I moved database (7.4.6 from fedora RPM) from the client machine to the virtual server using pg_dump and pg_restore over the Internet and modified update client to contact remote host instead of localhost. - After four months of operation on the virtual server the table "raakaleimat" - the one that the client mainly does inserts into - crashed with "Invalid page header". After some googling I decided to drop the table, recreate it, restore data from previous backup as inserts and restart the database - After four and half months of operation, a week ago, this happened again, and again I dropped the table, recreated it and restarted the database - Now another table "firmat" that holds keys almost every other table in the database crashed with "Invalid page header". And what is worse another non application related database has failed with "Invalid page header". It seems to me that the problem is missing log-file "/var/lib/pgsql/data/pg_clog/0000" which I failed to spot on 21st of April when the raakaleimat-table crashed second time. What can I do to recreate the missing file or is that what I should do? There are 13 databases on the system. Should I back them all up, drop everything. Remove every file, do initdb and restore everything? Should I kick the providers database rpm out of the box and fetch another one from somewhere else? Or should I downgrade client environment and recompile ? Logs: Apr 21 14:07:45 claymountain postgres[6453]: [8-1] FATAL: the database system is starting up Apr 21 14:07:45 claymountain postgres[5912]: [16-1] LOG: database system is ready Apr 21 14:08:35 claymountain postgres[7155]: [8-2] DETAIL: could not open file "/var/lib/pgsql/data/pg_clog/0295": No such file or directory Apr 21 14:10:02 claymountain postgres[11079]: [9-2] DETAIL: could not open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory Apr 21 14:11:11 claymountain postgres[22476]: [8-1] LOG: received fast shutdown request Apr 21 14:11:11 claymountain postgres[22476]: [9-1] LOG: aborting any active transactions Apr 21 14:11:11 claymountain postgres[12042]: [10-1] FATAL: terminating connection due to administrator command Apr 21 14:11:11 claymountain postgres[12016]: [10-1] FATAL: terminating connection due to administrator command Apr 21 14:11:21 claymountain postgres[13074]: [11-1] LOG: database system is shut down Apr 21 14:11:42 claymountain postgres[13614]: [1-1] LOG: could not create IPv6 socket: Address family not supported by protocol Apr 21 14:11:42 claymountain postgres[13618]: [2-1] LOG: database system was shut down at 2005-04-21 14:11:21 EEST Apr 21 14:11:42 claymountain postgres[13618]: [3-1] LOG: checkpoint record is at 6/350B0FDC Apr 21 14:11:42 claymountain postgres[13618]: [4-1] LOG: redo record is at 6/350B0FDC; undo record is at 0/0; shutdown TRUE Apr 21 14:11:42 claymountain postgres[13618]: [5-1] LOG: next transaction ID: 50072169; next OID: 3492415 Apr 21 14:11:42 claymountain postgres[13618]: [6-1] LOG: database system is ready Apr 21 14:13:38 claymountain postgres[15582]: [3-2] DETAIL: could not open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory ... Apr 28 08:08:03 claymountain postgres[3227]: [3-2] DETAIL: could not open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory Apr 28 07:59:33 claymountain postgres[25623]: [2-1] ERROR: invalid page header in block 1 of relation "firmat" This is on database pikkusisko, the one that has crashed twice before Apr 28 08:12:40 claymountain postgres[10835]: [2-1] LOG: recycled transaction log file "00000006000000B1" Apr 28 08:41:05 claymountain postgres[7602]: [4-1] ERROR: invalid page header in block 3 of relation "serendipity_suppress" This one is one of the four serendipity databases we have, I do not yet know which one Apr 28 08:46:45 claymountain postgres[14998]: [3-2] DETAIL: could not open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory Regards, Mauri Sahlberg ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Help desperately needed: reoccurring invalid page 2005-04-28 06:27 Help desperately needed: reoccurring invalid page header Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> @ 2005-04-28 12:58 ` Ian Westmacott <ianw@intellivid.com> 2005-04-29 06:38 ` Re: Help desperately needed: reoccurring invalid page Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> 0 siblings, 1 reply; 3+ messages in thread From: Ian Westmacott @ 2005-04-28 12:58 UTC (permalink / raw) To: Mauri Sahlberg <Mauri.Sahlberg@claymountain.com>; +Cc: pgsql-admin@postgresql.org, peikko@claymountain.com, PoRRo <theporro@saunalahti.fi> Mauri, I'm no expert but I can tell you what I know. We have experienced this same problem, and I asked a similar question not too long ago. - This is probably due to some sort of I/O problem. In our case, it occurs very frequently on reboots, and it appears to be related to dirty buffers not being flushed out when the partition is unmounted. Look into a newer kernel, and/or newer/different filesystem (we have seen this in various flavors of the 2.4.6 kernel with XFS, JFS and EXT3). - If you have invalid page headers, you should take a look at the run-time parameter zero_damaged_pages. You can use this to fix up a table (losing the data on the bad pages). - We have not found a way around the missing log file problem. This appears to end up locking some rows forever. Others on this list may be able to tell you about tools for examining/fixing corrupt tables and log files. --Ian ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Help desperately needed: reoccurring invalid page 2005-04-28 06:27 Help desperately needed: reoccurring invalid page header Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> 2005-04-28 12:58 ` Re: Help desperately needed: reoccurring invalid page Ian Westmacott <ianw@intellivid.com> @ 2005-04-29 06:38 ` Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> 0 siblings, 0 replies; 3+ messages in thread From: Mauri Sahlberg @ 2005-04-29 06:38 UTC (permalink / raw) To: Ian Westmacott <ianw@intellivid.com>; +Cc: pgsql-admin; theporro@saunalahti.fi; peikko@claymountain.com Hi, Ian Westmacott wrote: >Mauri, I'm no expert but I can tell you what I know. We >have experienced this same problem, and I asked a >similar question not too long ago. > >- This is probably due to some sort of I/O problem. In > our case, it occurs very frequently on reboots, and > it appears to be related to dirty buffers not being > flushed out when the partition is unmounted. Look > into a newer kernel, and/or newer/different > filesystem (we have seen this in various flavors > of the 2.4.6 kernel with XFS, JFS and EXT3). > > Uh. As our cheap server is a virtual machine under a virtuozzo (http://www.sw-soft.com/products/virtuozzo/) I have no control over what kernel to run or what filesystem to use. Nor has there been frequent reboots. I have restarted the virtual server twice in 4 months and the provider has done it once since production started. But as it might still be a filesystem problem (vzfs) I should try to find documentation on vzfs to see if I can do something to increase its "reliability" or if the provider can do that. The client side did fail very frequently for a while due non mature w-lan card driver and that might have caused the corruption that did not got fixed properly by my inept attempts and still lurks around. >- If you have invalid page headers, you should take a > look at the run-time parameter zero_damaged_pages. > You can use this to fix up a table (losing the data > on the bad pages). > > > Thanks I will try that. >- We have not found a way around the missing log file > problem. This appears to end up locking some rows > forever. > > It seems to me that the best course for action for us would be: Zeroing damaged pages. Dump every database excluding templates. Shutdown. Remove provider's postgresql-rpms and install the latest rpms on 7-series of Postgresql from Fedora legacy/pgfoundry. Restore databases. Manually merge changes from older backups that got zeroed. Upgrade client-side to the same version of postgresql as the server side. Recompile libraries and applications. Startup. Backup. Increase frequenzy of backups and hope for the best. ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2005-04-29 06:38 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2005-04-28 06:27 Help desperately needed: reoccurring invalid page header Mauri Sahlberg <Mauri.Sahlberg@claymountain.com> 2005-04-28 12:58 ` Re: Help desperately needed: reoccurring invalid page Ian Westmacott <ianw@intellivid.com> 2005-04-29 06:38 ` Re: Help desperately needed: reoccurring invalid page Mauri Sahlberg <Mauri.Sahlberg@claymountain.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox