pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedSevere Badness On My Server: psql: FATAL: the database system is starting up
3+ messages / 2 participants
[nested] [flat]
* Severe Badness On My Server: psql: FATAL: the database system is starting up
@ 2005-03-13 16:12 Mitchell Laks <mlaks@verizon.net>
2005-03-13 18:58 ` Re: Severe Badness On My Server: psql: FATAL: the database system is starting up Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 3+ messages in thread
From: Mitchell Laks @ 2005-03-13 16:12 UTC (permalink / raw)
To: pgsql-admin
Dear Gurus:
My Server and me have had a very bad weekend, starting Friday afternoon.
I am running Debian Sarge, Postgresql 7.4.6 with linux kernel 2.6.8.
I am running a Postgresql backed application on a remote server. The system
has a system drive, on which the Postgresql database runs and there is a raid
1 drive on which the application stores data.
Well, the raid1 failed (or is failing - or is trying its hardest to fail, not
clear yet...). This should not have affected the Postgresql database as it is
safely on a separate drive.
However, when i logged onto the system, I found that I could not turn off
postgresql. I logged in as postgres, did pg_ctl stop and it did ....... and
then could not stop (presumably because hanging client applications were not
loged off the database).
So then I killed all the application clients (kill -9 of them), and still I
tried to pg_ctl stop and it did not want to stop.
So I looked in ps aux and the client applications looked like they were in D
status in ps aux.
wustl 18232 0.0 0.2 4872 1920 ? D Mar11
0:00 /usr/local/ctn/bi
I then tried to reboot system remotely via login as root and shutdown -r now
and even shutdown -h now. Interestingly enough (I have never ever seen this -
system refused to shutdown!!!!!!!).
I was floored! Well what to do? I decided to sleep on it.
Well I logged in then on saturday night and system was still hanging in this
bizarre state. I now saw qued shutdown requests in the ps aux. And nothing
was happening fast.
I thought. I read a little. I tried pg_ctl stop -m fast. It did nothing. I
prayed. I tried to do pg_dump LTA_IDB >lta_idb.dump to dump the database in
question. It didnt do anything.
I was desparate. I decided to try desparate measures I then pulled the gun
pg_ctl stop -m i.
OK so it stopped. Then I said let me try to dump the database and so I did
pg_ctl start. It started
postgres@A1:~$ pg_ctl status
pg_ctl: postmaster is running (PID: 21195)
Command line was:
/usr/lib/postgresql/bin/postmaster
Then I tried to dump the database and i got some message about the fact that
Fatal the database was starting. I waited a while and then I tried again.
same message. I then tried as user of the database psql LTA_IDB and message
Fatal the database is starting.
Then I tried psql LTA_IDB and got Fatal database is starting.
I waited. Then I did pg_ctl stop (I dont know why i did it. Perversity I
think.)
It then said to me
................ something about unable to stop.
Then I did
postgres@A1:~$ pg_dump LTA_IDB>lta_idb.dump
2005-03-13 10:56:33 [21481] LOG: connection received: host=[local] port=
2005-03-13 10:56:33 [21481] FATAL: the database system is shutting down
pg_dump: [archiver (db)] connection to database "LTA_IDB" failed: FATAL: the
dn
Now I did
pg_ctl status
postgres@A1:~$ pg_ctl status
pg_ctl: postmaster is running (PID: 21195)
Command line was:
/usr/lib/postgresql/bin/postmaster
OK I feel like I am in the twilight zone.
Next I did as root
cd /var/log
ls postg*
A1:/var/log# ls post*
postgres.log postgres.log.2.gz postgres.log.5.gz postgres.log.8.gz
postgres.log.1 postgres.log.3.gz postgres.log.6.gz postgres.log.9.gz
postgres.log.10.gz postgres.log.4.gz postgres.log.7.gz
A1:/var/log# less postgres.log
postgres.log: No such file or directory
WHAT????????
df -h
/dev/sda2 9.2G 2.8G 6.0G 32% /
tmpfs 443M 0 443M 0% /dev/shm
/dev/sda1 89M 11M 74M 13% /boot
/dev/sda3 7.4G 273M 6.7G 4% /home
/dev/sda8 11G 33M 9.9G 1% /mirror
/dev/sda7 449M 8.1M 417M 2% /tmp
/dev/sda6 7.4G 4.7G 2.4G 67% /var
/dev/md0 230G 139G 80G 64% /home/big0
I am in the twilight zone. My sanity is suspect. Any ideas on what to do next?
Pull the plug????
Mitchell
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Severe Badness On My Server: psql: FATAL: the database system is starting up
2005-03-13 16:12 Severe Badness On My Server: psql: FATAL: the database system is starting up Mitchell Laks <mlaks@verizon.net>
@ 2005-03-13 18:58 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 3+ messages in thread
From: Tom Lane @ 2005-03-13 18:58 UTC (permalink / raw)
To: Mitchell Laks <mlaks@verizon.net>; +Cc: pgsql-admin
Mitchell Laks <mlaks@verizon.net> writes:
> Well, the raid1 failed (or is failing - or is trying its hardest to fail, not
> clear yet...). This should not have affected the Postgresql database as it is
> safely on a separate drive.
Try turning off the power, physically disconnecting the raid1, and rebooting.
It sounds to me like the raid drive is so wedged that the kernel is
getting confused (or at least hanging operations that theoretically
shouldn't hang).
regards, tom lane
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Severe Badness On My Server: psql: FATAL: the database
@ 2005-03-14 07:53 Mitchell Laks <mlaks@verizon.net>
0 siblings, 0 replies; 3+ messages in thread
From: Mitchell Laks @ 2005-03-14 07:53 UTC (permalink / raw)
To: pgsql-admin
Thank you Tom and Frank!
As usual you gurus were 100% on target.
I had a raid1 failure on my data drive which caused the /var partition on my
system drive to fill with kernel messages in /var/log/messages from the
failing /dev/hdg1 drive.
Therefore, when I logged in remotely and tried to work with the system, I
first made room in /var by deleting stuff from /var/log/messages. But I didnt
want to power down the machine cause it is in a different borough here in
NYC :). I wanted to reboot remotely, which I could not do cause /var filling
must of messed up other processes which refused to allow me to reboot via
software.
So finally I pushed the button. I drove over and rebooted and removed /dev/md0
from the /etc/fstab and booted into linux, pg_dumped the database and moved
it to another server and all is fine.
I suspect I should move /var/lib/postgresql to its own partition to separate
out the point of failure of the hard drives from postgresql. What do you
think?
Thanks again both of you!,
Mitchell Laks
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2005-03-14 07:53 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-03-13 16:12 Severe Badness On My Server: psql: FATAL: the database system is starting up Mitchell Laks <mlaks@verizon.net>
2005-03-13 18:58 ` Re: Severe Badness On My Server: psql: FATAL: the database system is starting up Tom Lane <tgl@sss.pgh.pa.us>
2005-03-14 07:53 Re: Severe Badness On My Server: psql: FATAL: the database Mitchell Laks <mlaks@verizon.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox