pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedhung postmaster when client machine dies?
5+ messages / 5 participants
[nested] [flat]
* hung postmaster when client machine dies?
@ 2004-01-30 01:00 Mark Harrison <mh@pixar.com>
0 siblings, 1 reply; 5+ messages in thread
From: Mark Harrison @ 2004-01-30 01:00 UTC (permalink / raw)
To: pgsql-admin
We recently had an incident where a linux box went down with
a kernel error. A process on the box had an open connection
to a postgres session, and was in a transaction.
This situation was noticed when other processes connected to
postgres would not respond.
We observed that there was postmaster connected to the ip address
of the downed machine with an "in transaction" status. Killing
that processes unblocked the other processes.
Is this expected behavior? Was postgres simply waiting for
a failure from the TCP/IP layer?
We're now running a watchdog process that pings machines for
which a postmaster is running and killing that process if
the machine is not contactable for a certain period of time.
Thanks to whoever made the status information show up in ps
output!
Unfortunately, we didn't capture the process data... if this
would help we can attempt to reproduce the situation.
Many TIA,
Mark
--
Mark Harrison
Pixar Animation Studios
Emeryville, CA
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: hung postmaster when client machine dies?
@ 2004-01-30 15:18 Jeff <threshar@torgo.978.org>
parent: Mark Harrison <mh@pixar.com>
0 siblings, 1 reply; 5+ messages in thread
From: Jeff @ 2004-01-30 15:18 UTC (permalink / raw)
To: Mark Harrison <mh@pixar.com>; +Cc: pgsql-admin
On Thu, 29 Jan 2004 17:00:27 -0800
Mark Harrison <mh@pixar.com> wrote:
> We observed that there was postmaster connected to the ip address
> of the downed machine with an "in transaction" status. Killing
> that processes unblocked the other processes.
>
> Is this expected behavior? Was postgres simply waiting for
> a failure from the TCP/IP layer?
>
When a machine simply "goes away" (crashed, unplugged) no packets are
sent indicating the socket has closed so PG doesn't know the socket is
closed and cannot rollback the transaction. (This is true of any
tcp-based protocol).
If that connection had a transaction open it'll likely have rows locked.
Killing PG caused it to rollback that transaction freeing those locks.
Eventually PG would have found out the socket was dead. Most likely if
it tried to write to it.
--
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: hung postmaster when client machine dies?
@ 2004-01-30 15:42 Tom Lane <tgl@sss.pgh.pa.us>
parent: Jeff <threshar@torgo.978.org>
0 siblings, 0 replies; 5+ messages in thread
From: Tom Lane @ 2004-01-30 15:42 UTC (permalink / raw)
To: Jeff <threshar@torgo.978.org>; +Cc: Mark Harrison <mh@pixar.com>; pgsql-admin
Jeff <threshar@torgo.978.org> writes:
> Eventually PG would have found out the socket was dead.
We do enable TCP keepalive on the client socket, so eventually the
kernel will time out and notify us of a lost connection. Unfortunately
the timeouts involved are long --- IIRC the relevant RFCs specify at
least an hour.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: hung postmaster when client machine dies?
@ 2004-01-30 15:42 Goulet, Dick <DGoulet@vicr.com>
0 siblings, 1 reply; 5+ messages in thread
From: Goulet, Dick @ 2004-01-30 15:42 UTC (permalink / raw)
To: pgsql-admin
Hope you don't mind if I disagree. Most OS's that have a tcp/ip layer also have a parameter therein called tcp_keep_alive. They also set this parameter to infinity. The purpose of tcp_keep_alive is to have the OS kernel periodically verify that all tcp/ip connections it is managing are still functioning every so often. Basically the OS sends a probe packet down the line to the to the client machine. If it bounces back the connection is dead & the OS can do what it has to, which will then notify postmaster just like you manually did. I'd contact your OS vendor for information of what tcp_keep_alive is set to by default and how you can change it.
Dick Goulet
Senior Oracle DBA
Oracle Certified 8i DBA
-----Original Message-----
From: Jeff [mailto:threshar@torgo.978.org]
Sent: Friday, January 30, 2004 10:18 AM
To: Mark Harrison
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] hung postmaster when client machine dies?
On Thu, 29 Jan 2004 17:00:27 -0800
Mark Harrison <mh@pixar.com> wrote:
> We observed that there was postmaster connected to the ip address
> of the downed machine with an "in transaction" status. Killing
> that processes unblocked the other processes.
>
> Is this expected behavior? Was postgres simply waiting for
> a failure from the TCP/IP layer?
>
When a machine simply "goes away" (crashed, unplugged) no packets are
sent indicating the socket has closed so PG doesn't know the socket is
closed and cannot rollback the transaction. (This is true of any
tcp-based protocol).
If that connection had a transaction open it'll likely have rows locked.
Killing PG caused it to rollback that transaction freeing those locks.
Eventually PG would have found out the socket was dead. Most likely if
it tried to write to it.
--
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: hung postmaster when client machine dies?
@ 2004-02-07 10:54 Peter Galbavy <peter.galbavy@knowtion.net>
parent: Goulet, Dick <DGoulet@vicr.com>
0 siblings, 0 replies; 5+ messages in thread
From: Peter Galbavy @ 2004-02-07 10:54 UTC (permalink / raw)
To: Goulet, Dick <DGoulet@vicr.com>; pgsql-admin
Goulet, Dick wrote:
> Hope you don't mind if I disagree. Most OS's that have a tcp/ip
> layer also have a parameter therein called tcp_keep_alive. They also
> set this parameter to infinity. The purpose of tcp_keep_alive is to
> have the OS kernel periodically verify that all tcp/ip connections it
> is managing are still functioning every so often. Basically the OS
> sends a probe packet down the line to the to the client machine. If
> it bounces back the connection is dead & the OS can do what it has
> to, which will then notify postmaster just like you manually did.
> I'd contact your OS vendor for information of what tcp_keep_alive is
> set to by default and how you can change it.
On the other hand I suggest that you *do not* change this value, even if you
know how, without being very careful understanding what it means.
Changing the system-wide TCP KA time can have unforseen effects on other,
unrelated functions of the systems concerned.
This is simply why most long lived protocols layered over TCP also tend to
have their own keep alive PDUs.
Peter
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2004-02-07 10:54 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-01-30 01:00 hung postmaster when client machine dies? Mark Harrison <mh@pixar.com>
2004-01-30 15:18 ` Jeff <threshar@torgo.978.org>
2004-01-30 15:42 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-01-30 15:42 Re: hung postmaster when client machine dies? Goulet, Dick <DGoulet@vicr.com>
2004-02-07 10:54 ` Peter Galbavy <peter.galbavy@knowtion.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox