pg.ddx.io pgsql-admin@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Possible database corruption - urgent 4+ messages / 2 participants [nested] [flat]
* Possible database corruption - urgent @ 2013-01-07 21:22 Benjamin Krajmalnik <kraj@servoyant.com> 0 siblings, 0 replies; 4+ messages in thread From: Benjamin Krajmalnik @ 2013-01-07 21:22 UTC (permalink / raw) To: pgsql-admin I have a situation where pg_xlog started growing until it filled up the disk drive. I got alerted to the error and started investigating. Checked the logs and I am seeing the following entry repeatedly: 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm I checked the actual file system, and that file is indeed missing. 181979366 exists. Is there a way to get the system back up and running? I stopped the postmaster and am moving the pg_xlog directory to a partition which has room left in it, but I need to resolve this missing file problem ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Possible database corruption - urgent @ 2013-01-07 21:31 Benjamin Krajmalnik <kraj@servoyant.com> 0 siblings, 0 replies; 4+ messages in thread From: Benjamin Krajmalnik @ 2013-01-07 21:31 UTC (permalink / raw) To: Benjamin Krajmalnik <kraj@servoyant.com>; pgsql-admin I forgot to mention - PostgreSQL 9.0 - my apologies. Can I just recreate the file using touch so it exists and then restart potgresql? The system coredumped and was attempting to go intorecovery mode 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm . . . 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Benjamin Krajmalnik Sent: Monday, January 07, 2013 2:22 PM To: pgsql-admin@postgresql.org Subject: [ADMIN] Possible database corruption - urgent I have a situation where pg_xlog started growing until it filled up the disk drive. I got alerted to the error and started investigating. Checked the logs and I am seeing the following entry repeatedly: 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm I checked the actual file system, and that file is indeed missing. 181979366 exists. Is there a way to get the system back up and running? I stopped the postmaster and am moving the pg_xlog directory to a partition which has room left in it, but I need to resolve this missing file problem ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Possible database corruption - urgent @ 2013-01-07 21:35 Benjamin Krajmalnik <kraj@servoyant.com> 2013-01-07 22:21 ` Re: Possible database corruption - urgent Walter Hurry <walterhurry@gmail.com> 0 siblings, 1 reply; 4+ messages in thread From: Benjamin Krajmalnik @ 2013-01-07 21:35 UTC (permalink / raw) To: pgsql-admin Sorry for the cut and paste error. This is the log entry when the pg_xlog partition ran out of space: 2013-01-07 20:50:22 GMT [local]PANIC: could not write to file "pg_xlog/xlogtemp.49680": No space left on device 2013-01-07 20:50:22 GMT [local]STATEMENT: INSERT INTO tbltmptests (testhash, testtime, statusid, replytxt, replyval, groupid) V 2013-01-07 20:50:23 GMT LOG: server process (PID 49680) was terminated by signal 6: Abort trap 2013-01-07 20:50:23 GMT LOG: terminating any other active server processes 2013-01-07 20:50:23 GMT [local]WARNING: terminating connection because of crash of another server process 2013-01-07 20:50:23 GMT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction an 2013-01-07 20:50:23 GMT [local]HINT: In a moment you should be able to reconnect to the database and repeat your command. . . . 2013-01-07 20:50:23 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:50:23 GMT LOG: all server processes terminated; reinitializing 2013-01-07 20:50:24 GMT LOG: database system was interrupted; last known up at 2013-01-07 00:31:02 GMT 2013-01-07 20:50:24 GMT LOG: database system was not properly shut down; automatic recovery in progress 2013-01-07 20:50:24 GMT LOG: consistent recovery state reached at 52F/8CE57490 2013-01-07 20:50:24 GMT LOG: redo starts at 52F/7BABC118 2013-01-07 20:50:38 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:50:53 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:51:08 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:51:24 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:51:39 GMT [local]FATAL: the database system is in recovery mode 2013-01-07 20:51:54 GMT [local]FATAL: the database system is in recovery mode From: Benjamin Krajmalnik Sent: Monday, January 07, 2013 2:31 PM To: Benjamin Krajmalnik; pgsql-admin@postgresql.org Subject: RE: [ADMIN] Possible database corruption - urgent I forgot to mention - PostgreSQL 9.0 - my apologies. Can I just recreate the file using touch so it exists and then restart potgresql? The system coredumped and was attempting to go intorecovery mode 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm . . . 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Benjamin Krajmalnik Sent: Monday, January 07, 2013 2:22 PM To: pgsql-admin@postgresql.org Subject: [ADMIN] Possible database corruption - urgent I have a situation where pg_xlog started growing until it filled up the disk drive. I got alerted to the error and started investigating. Checked the logs and I am seeing the following entry repeatedly: 2013-01-07 01:49:12 GMT ERROR: could not open file "base/16748/181979366_fsm": No such file or directory 2013-01-07 01:49:12 GMT CONTEXT: writing block 1 of relation base/16748/181979366_fsm 2013-01-07 01:49:12 GMT WARNING: could not write block 1 of base/16748/181979366_fsm I checked the actual file system, and that file is indeed missing. 181979366 exists. Is there a way to get the system back up and running? I stopped the postmaster and am moving the pg_xlog directory to a partition which has room left in it, but I need to resolve this missing file problem ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Possible database corruption - urgent 2013-01-07 21:35 Re: Possible database corruption - urgent Benjamin Krajmalnik <kraj@servoyant.com> @ 2013-01-07 22:21 ` Walter Hurry <walterhurry@gmail.com> 0 siblings, 0 replies; 4+ messages in thread From: Walter Hurry @ 2013-01-07 22:21 UTC (permalink / raw) To: pgsql-admin On Mon, 07 Jan 2013 14:35:48 -0700, Benjamin Krajmalnik wrote: <snip> What do you think you will gain by adding "urgent" to your subject line? What do you think you will gain by posting in HTML? -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2013-01-07 22:21 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2013-01-07 21:22 Possible database corruption - urgent Benjamin Krajmalnik <kraj@servoyant.com> 2013-01-07 21:31 Re: Possible database corruption - urgent Benjamin Krajmalnik <kraj@servoyant.com> 2013-01-07 21:35 Re: Possible database corruption - urgent Benjamin Krajmalnik <kraj@servoyant.com> 2013-01-07 22:21 ` Walter Hurry <walterhurry@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox