From: Craig A. James <cjames@modgraph-usa.com>
To: pgsql-performance@postgresql.org
Subject: Postgres server crash
Date: Wed, 15 Nov 2006 18:20:24 -0800
Message-ID: <455BCAE8.8010905@modgraph-usa.com> (raw)
In-Reply-To: <453EF7C4.2040105@modgraph-usa.com>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
<b42b73150610201811m84369f7kc7d58b39aed3ad73@mail.gmail.com>
<ce4072df0610230810t5d66bbb8xf1e9c0ed71b6b6c0@mail.gmail.com>
<20061023215913.GW26892@nasby.net>
<ce4072df0610240617s72f35540odd3d4e0903313b63@mail.gmail.com>
<20061025001709.GA26892@nasby.net>
<453EF7C4.2040105@modgraph-usa.com>
For the third time today, our server has crashed, or frozen, actually something in between. Normally there are about 30-50 connections because of mod_perl processes that keep connections open. After the crash, there are three processes remaining:
# ps -ef | grep postgres
postgres 23832 1 0 Nov11 pts/1 00:02:53 /usr/local/pgsql/bin/postmaster -D /postgres/main
postgres 1200 23832 20 14:28 pts/1 00:58:14 postgres: pubchem pubchem 66.226.76.106(58882) SELECT
postgres 4190 23832 25 14:33 pts/1 01:09:12 postgres: asinex asinex 66.226.76.106(56298) SELECT
But they're not doing anything: No CPU time consumed, no I/O going on, no progress. If I try to connect with psql(1), it says:
psql: FATAL: the database system is in recovery mode
And the server log has:
LOG: background writer process (PID 23874) was terminated by signal 9
LOG: terminating any other active server processes
LOG: statistics collector process (PID 23875) was terminated by signal 9
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited ab
normally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited ab
... repeats about 50 times, one per process.
Questions:
1. Any idea what happened and how I can avoid this? It's a *big* problem.
2. Why didn't the database recover? Why are there two processes
that couldn't be killed?
3. Where did the "signal 9" come from? (Nobody but me ever logs
in to the server machine.)
Help!
Thanks,
Craig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: cjames@modgraph-usa.com
Subject: Re: Postgres server crash
In-Reply-To: <455BCAE8.8010905@modgraph-usa.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox