pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedstrange hung processes
6+ messages / 4 participants
[nested] [flat]
* strange hung processes
@ 2006-02-02 17:11 Jeremy Ashcraft <jashcraft@edgate.com>
0 siblings, 2 replies; 6+ messages in thread
From: Jeremy Ashcraft @ 2006-02-02 17:11 UTC (permalink / raw)
To: pgsql-admin
We're running 7.4.9 and have run into something strange.
A group of processes seem to have hung and cannot be killed. netstat
showed that there were no active TCP connections at the time. Sending
SIGTERM to the parent process caused PG to begin its shutdown, but it
never finished. We then kill -9 the postmaster, which caused it to die,
but these child procs remain, even after a clean restart. Our next step
is to reboot the box, but before we do, I'm just curious anyone else has
seen this, why this happened, and if anyone know ways to prevent this
from happening again.....
thanks
--
jeremy ashcraft
operations/development
EDucation GATEways
jashcraft@edgate.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: strange hung processes
@ 2006-02-02 17:32 Jeremy Ashcraft <jashcraft@edgate.com>
parent: Jeremy Ashcraft <jashcraft@edgate.com>
1 sibling, 0 replies; 6+ messages in thread
From: Jeremy Ashcraft @ 2006-02-02 17:32 UTC (permalink / raw)
To: pgsql-admin
I forgot to include the process list:
postgres 13214 5142 0 Jan09 ? 00:00:00 postgres: postgre edgate
10.1.1.3 authentication
postgres 13215 5142 0 Jan09 ? 00:00:00 postgres: postgre edgate
10.1.1.1 authentication
postgres 13216 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
postgres 13217 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13218 5142 0 Jan09 ? 00:00:00 postgres: postgre edgate
10.1.1.1 authentication
postgres 13219 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13220 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13221 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13222 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
postgres 13223 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13224 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
postgres 13225 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
postgres 13226 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13227 5142 0 Jan09 ? 00:00:00 postgres: snuser
sn_master 10.1.1.3 authentication
postgres 13228 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
postgres 13229 5142 0 Jan09 ? 00:00:00 postgres: postgre
sn_master 10.1.1.3 authentication
Jeremy Ashcraft wrote:
> We're running 7.4.9 and have run into something strange.
> A group of processes seem to have hung and cannot be killed. netstat
> showed that there were no active TCP connections at the time. Sending
> SIGTERM to the parent process caused PG to begin its shutdown, but it
> never finished. We then kill -9 the postmaster, which caused it to
> die, but these child procs remain, even after a clean restart. Our
> next step is to reboot the box, but before we do, I'm just curious
> anyone else has seen this, why this happened, and if anyone know ways
> to prevent this from happening again.....
>
> thanks
>
--
jeremy ashcraft
operations/development
EDucation GATEways
jashcraft@edgate.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: strange hung processes
@ 2006-02-02 18:40 Ben Kim <bkim@coe.tamu.edu>
parent: Jeremy Ashcraft <jashcraft@edgate.com>
1 sibling, 0 replies; 6+ messages in thread
From: Ben Kim @ 2006-02-02 18:40 UTC (permalink / raw)
To: Jeremy Ashcraft <jashcraft@edgate.com>; +Cc: pgsql-admin
>We're running 7.4.9 and have run into something strange.
>A group of processes seem to have hung and cannot be killed. netstat
>showed that there were no active TCP connections at the time. Sending
>SIGTERM to the parent process caused PG to begin its shutdown, but it
>never finished. We then kill -9 the postmaster, which caused it to die,
>but these child procs remain, even after a clean restart. Our next step
>is to reboot the box, but before we do, I'm just curious anyone else has
>seen this, why this happened, and if anyone know ways to prevent this
>from happening again.....
If it's not sensitive information, what does this show?
lsof | grep 'pid of hung process'
Regards,
Ben Kim
Developer
http://benix.tamu.edu
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: strange hung processes
@ 2006-02-02 19:39 Larry Rosenman <lrosenman@pervasive.com>
0 siblings, 1 reply; 6+ messages in thread
From: Larry Rosenman @ 2006-02-02 19:39 UTC (permalink / raw)
To: Ben Kim <bkim@coe.tamu.edu>; Jeremy Ashcraft <jashcraft@edgate.com>; +Cc: pgsql-admin
Ben Kim wrote:
>> We're running 7.4.9 and have run into something strange.
>> A group of processes seem to have hung and cannot be killed. netstat
>> showed that there were no active TCP connections at the time.
>> Sending SIGTERM to the parent process caused PG to begin its
>> shutdown, but it never finished. We then kill -9 the postmaster,
>> which caused it to die, but these child procs remain, even after a
>> clean restart. Our next step is to reboot the box, but before we
>> do, I'm just curious anyone else has seen this, why this happened,
>> and if anyone know ways to prevent this from happening again.....
>
> If it's not sensitive information, what does this show?
>
> lsof | grep 'pid of hung process'
>
from the save-a-process committee:
lsof -p 'pid of hung process'
will give the same info :)
LER
--
Larry Rosenman
Database Support Engineer
PERVASIVE SOFTWARE. INC.
12365B RIATA TRACE PKWY
3015
AUSTIN TX 78727-6531
Tel: 512.231.6173
Fax: 512.231.6597
Email: Larry.Rosenman@pervasive.com
Web: www.pervasive.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: strange hung processes
@ 2006-02-02 20:22 Jeremy Ashcraft <jashcraft@edgate.com>
parent: Larry Rosenman <lrosenman@pervasive.com>
0 siblings, 1 reply; 6+ messages in thread
From: Jeremy Ashcraft @ 2006-02-02 20:22 UTC (permalink / raw)
To: pgsql-admin
If it's not sensitive information, what does this show?
>> lsof | grep 'pid of hung process'
>>
From my systems guy:
in /proc
codd 13262 # cat status {proc "filesystem"}
Name: postmaster
State: D (disk sleep)
Because of the "D" state, it can't be killed as it is not interuptible
(waiting on IO ?).
codd opt # lsof -p 13263
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
postmaste 13263 postgres cwd DIR 8,17 464 4030
/app1/user/postgres
postmaste 13263 postgres rtd DIR 8,3 488 2 /
postmaste 13263 postgres txt REG 8,17 2552859 5012
/app1/pg/7.4.9/bin/postgres
postmaste 13263 postgres mem REG 0,0 0 [heap]
(stat: No such file or directory)
postmaste 13263 postgres DEL REG 0,7 0 /SYSV0052e2c1
postmaste 13263 postgres mem REG 8,3 35428 5729
/lib/libnss_files-2.3.4.so
postmaste 13263 postgres mem REG 8,5 18852 6895
/usr/lib/libgpm.so.1.19.0
postmaste 13263 postgres mem REG 8,3 266284 5965
/lib/libncurses.so.5.4
postmaste 13263 postgres mem REG 8,3 1167808 5977
/lib/libc-2.3.4.so
postmaste 13263 postgres mem REG 8,3 151896 5755
/lib/libm-2.3.4.so
postmaste 13263 postgres mem REG 8,3 10620 5987
/lib/libdl-2.3.4.so
postmaste 13263 postgres mem REG 8,3 75848 6003
/lib/libnsl-2.3.4.so
postmaste 13263 postgres mem REG 8,3 60884 5998
/lib/libresolv-2.3.4.so
postmaste 13263 postgres mem REG 8,3 18424 6006
/lib/libcrypt-2.3.4.so
postmaste 13263 postgres mem REG 8,3 173996 6007
/lib/libreadline.so.4.3
postmaste 13263 postgres mem REG 8,3 63204 5976
/lib/libz.so.1.2.2
postmaste 13263 postgres mem REG 8,3 95392 5915
/lib/ld-2.3.4.so
postmaste 13263 postgres 0r CHR 1,3 2767 /dev/null
postmaste 13263 postgres 1w REG 8,7 7513385 3649
/var/log/pg/ruby.log.bak1
postmaste 13263 postgres 2w REG 8,7 7513385 3649
/var/log/pg/ruby.log.bak1
postmaste 13263 postgres 5u sock 0,4 11790654 can't
identify protocol
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: strange hung processes
@ 2006-02-03 04:47 Tom Lane <tgl@sss.pgh.pa.us>
parent: Jeremy Ashcraft <jashcraft@edgate.com>
0 siblings, 0 replies; 6+ messages in thread
From: Tom Lane @ 2006-02-03 04:47 UTC (permalink / raw)
To: Jeremy Ashcraft <jashcraft@edgate.com>; +Cc: pgsql-admin
Jeremy Ashcraft <jashcraft@edgate.com> writes:
> in /proc
> codd 13262 # cat status {proc "filesystem"}
> Name: postmaster
> State: D (disk sleep)
> Because of the "D" state, it can't be killed as it is not interuptible
> (waiting on IO ?).
If the process is stuck in D state then it's not Postgres' fault.
You're looking at a hardware problem, or if the database is mounted
via NFS then it might be an NFS-protocol-level problem. In any case
you need to call out the kernel and hardware troops, not us database
weenies ...
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2006-02-03 04:47 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-02-02 17:11 strange hung processes Jeremy Ashcraft <jashcraft@edgate.com>
2006-02-02 17:32 ` Jeremy Ashcraft <jashcraft@edgate.com>
2006-02-02 18:40 ` Ben Kim <bkim@coe.tamu.edu>
2006-02-02 19:39 Re: strange hung processes Larry Rosenman <lrosenman@pervasive.com>
2006-02-02 20:22 ` Jeremy Ashcraft <jashcraft@edgate.com>
2006-02-03 04:47 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox