Received: from localhost (maia-2.hub.org [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 08AC89FB832 for ; Thu, 31 May 2007 13:55:33 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 90359-01 for ; Thu, 31 May 2007 13:55:20 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.177]) by postgresql.org (Postfix) with ESMTP id 6DC9D9FB778 for ; Thu, 31 May 2007 13:55:21 -0300 (ADT) Received: by wa-out-1112.google.com with SMTP id v27so302093wah for ; Thu, 31 May 2007 09:55:20 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=J2HVvonB/4pGXjTea5jlPHYUnq7o8huSfVjvALflZOXN77louTM9XljN2sHXut26SlbTjSwrZ2RBzFCC+3RQFLu0jZDAxL1ScJd0aA9bslKmTjUBAJPW6OF6StFFqOgEaAIzqXQ3VOblhi/joE8ewEsiqgSuwgeCNAEyv9ZYP3g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=K18pk9Jo37wFuDgR2XIKGK97hdKTLutXaXYn0GONnOPdv5trGOEdskgWTzXz1ajF3nnSM+zFh8YbcAh8QhvoOmkkQ/b3KGAMVKI9SuLel5Z6UGkgj63TixTVENWcouSSE8pUqgVsME1qkYZXKGxN7ls6vCJj2mgFO4D2BkK9xV4= Received: by 10.114.79.1 with SMTP id c1mr798722wab.1180630520287; Thu, 31 May 2007 09:55:20 -0700 (PDT) Received: by 10.114.197.11 with HTTP; Thu, 31 May 2007 09:55:20 -0700 (PDT) Message-ID: <4544e0330705310955g3c95679vbbda98773a687d22@mail.gmail.com> Date: Thu, 31 May 2007 11:55:20 -0500 From: "Peter Koczan" To: "Tom Lane" Subject: Re: trouble restarting a server Cc: pgsql-admin In-Reply-To: <4544e0330705221405w5c5a5eb2n5df936e7987cae97@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_23756_13466382.1180630520238" References: <4544e0330705211457k30c0cd40oec47e830a0791a9d@mail.gmail.com> <192.1179790145@sss.pgh.pa.us> <4544e0330705221405w5c5a5eb2n5df936e7987cae97@mail.gmail.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200705/339 X-Sequence-Number: 25659 ------=_Part_23756_13466382.1180630520238 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline It finally reoccurred. Here's what I got from attaching to those processes from gdb. I attached with the postmaster binary, let me know if I should use something else. vero(su): ps axvw | grep notify 24556 ? Ss 0:03 0 3265 41262 29672 0.7 postgres: jerel csdb chef(36275) notify interrupt 2889 ? Ss 0:04 0 3265 41270 29688 0.7 postgres: ela csdb newton(32777) notify interrupt 2943 ? Ss 0:04 0 3265 41270 29684 0.7 postgres: stefan csdb stupid(32788) notify interrupt 5866 ? Ss 0:04 0 3265 41270 29680 0.7 postgres: petska csdb brian(32786) notify interrupt 27850 ? Ss 0:03 0 3265 41270 29768 0.7 postgres: dparter csdb yfandes(35456) notify interrupt 18582 ? Ss 0:03 0 3265 41270 29732 0.7 postgres: timc csdb tornado(47047) notify interrupt 449 ? Ss 0:02 0 3265 41270 29764 0.7 postgres: archer csdb spoon(33141) notify interrupt 12731 pts/0 S+ 0:00 0 71 3828 664 0.0 grep notify vero(su): gdb /s/postgresql/bin/postmaster 24556 [----- Begin lots of messages -----] GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /afs/cs.wisc.edu/s/postgresql-8.2.4/@sys/bin/postmaster, process 5866 Reading symbols from /lib/libpam.so.0...done. Loaded symbols for /lib/libpam.so.0 Reading symbols from /afs/cs.wisc.edu/s/openssl-0.9.8d /@sys/lib/libssl.so.0.9.8d...done. Loaded symbols for /s/openssl-0.9.8d/lib/libssl.so.0.9.8d Reading symbols from /afs/cs.wisc.edu/s/openssl-0.9.8d /@sys/lib/libcrypto.so.0.9.8d...done. Loaded symbols for /s/openssl-0.9.8d/lib/libcrypto.so.0.9.8d Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1/@sys/lib/libkrb5.so.3...done.Loaded symbols for /s/krb5-1.5.1/lib/libkrb5.so.3 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1 /@sys/lib/libcom_err.so.3...done. Loaded symbols for /s/krb5-1.5.1/lib/libcom_err.so.3 Reading symbols from /lib/libaudit.so.0...done. Loaded symbols for /lib/libaudit.so.0 Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1 /i386_cent40/lib/libk5crypto.so.3...done. Loaded symbols for /s/krb5-1.5.1/i386_cent40/lib/libk5crypto.so.3 Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1 /i386_cent40/lib/libkrb5support.so.0...done. Loaded symbols for /s/krb5-1.5.1/i386_cent40/lib/libkrb5support.so.0 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...done. Loaded symbols for /lib/libnss_dns.so.2 0x007ef7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 [----- End lots of messages -----] (gdb) bt #0 0x007ef7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x008c60f3 in __write_nocancel () from /lib/tls/libc.so.6 #2 0x0064d734 in sock_write () from /s/openssl-0.9.8d /lib/libcrypto.so.0.9.8d #3 0x00000008 in ?? () #4 0x09ede1a2 in ?? () #5 0x00000038 in ?? () #6 0x09ee50c0 in ?? () #7 0x006e06c8 in ?? () from /s/openssl-0.9.8d/lib/libcrypto.so.0.9.8d #8 0x09ed4148 in ?? () #9 0x00000000 in ?? () All the other processes are the same except for addresses in #4, #6, and #8, but they're all within a few MB of each other (they're probably asynchronous interrupts). Let me know if you need more info. Peter On 5/22/07, Peter Koczan wrote: > > The release is 8.2.4. I haven't been able to reproduce the condition yet, > but I will send along stack traces as soon as I can. I have this strange > feeling that it's only going to happen when I find a reason to make a > restart-worthy config change. > > Peter > > On 5/21/07, Tom Lane < tgl@sss.pgh.pa.us> wrote: > > > > "Peter Koczan" writes: > > > [ lots of processes stuck in "notify interrupt" code ] > > > > That's weird. If it's still in that state, or if you can reproduce it, > > could you attach to a few of those processes with gdb and get stack > > traces? > > > > Looking at the async.c code, an obvious candidate is that that routine > > tries to take ExclusiveLock on pg_listener --- so if something had > > managed to exit without releasing a lock on that table, hangups could be > > expected. But if that were the case, you'd think the process status > > lines would include "waiting". My guess is they're blocked on something > > > > lower-level than a table lock, but without a stack trace it's hard to > > guess what. > > > > Which PG release is this exactly? > > > > regards, tom lane > > > > ------=_Part_23756_13466382.1180630520238 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline It finally reoccurred. Here's what I got from attaching to those processes from gdb. I attached with the postmaster binary, let me know if I should use something else.

vero(su): ps axvw | grep notify
24556 ?        Ss     0:03      0  3265 41262 29672  0.7 postgres: jerel csdb chef(36275) notify interrupt
 2889 ?        Ss     0:04      0  3265 41270 29688  0.7 postgres: ela csdb newton(32777) notify interrupt
 2943 ?        Ss     0:04      0  3265 41270 29684  0.7 postgres: stefan csdb stupid(32788) notify interrupt
 5866 ?        Ss     0:04      0  3265 41270 29680  0.7 postgres: petska csdb brian(32786) notify interrupt
27850 ?        Ss     0:03      0  3265 41270 29768  0.7 postgres: dparter csdb yfandes(35456) notify interrupt
18582 ?        Ss     0:03      0  3265 41270 29732  0.7 postgres: timc csdb tornado(47047) notify interrupt
  449 ?        Ss     0:02      0  3265 41270 29764  0.7 postgres: archer csdb spoon(33141) notify interrupt
12731 pts/0    S+     0:00      0    71  3828  664  0.0 grep notify

vero(su): gdb /s/postgresql/bin/postmaster 24556
[-----  Begin lots of messages -----]
GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Attaching to program: /afs/cs.wisc.edu/s/postgresql-8.2.4/@sys/bin/postmaster, process 5866
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /afs/cs.wisc.edu/s/openssl- 0.9.8d/@sys/lib/libssl.so.0.9.8d...done.
Loaded symbols for /s/openssl-0.9.8d/lib/libssl.so.0.9.8d
Reading symbols from /afs/cs.wisc.edu/s/openssl-0.9.8d/@sys/lib/libcrypto.so.0.9.8d...done.
Loaded symbols for /s/openssl- 0.9.8d/lib/libcrypto.so.0.9.8d
Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1/@sys/lib/libkrb5.so.3...done.Loaded symbols for /s/krb5-1.5.1/lib/libkrb5.so.3
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1/@sys/lib/libcom_err.so.3...done.
Loaded symbols for /s/krb5-1.5.1/lib/libcom_err.so.3
Reading symbols from /lib/libaudit.so.0...done.
Loaded symbols for /lib/libaudit.so.0
Reading symbols from /afs/cs.wisc.edu/s/krb5-1.5.1/i386_cent40/lib/libk5crypto.so.3...done.
Loaded symbols for /s/krb5-1.5.1/i386_cent40/lib/libk5crypto.so.3
Reading symbols from /afs/cs.wisc.edu/s/krb5- 1.5.1/i386_cent40/lib/libkrb5support.so.0...done.
Loaded symbols for /s/krb5-1.5.1/i386_cent40/lib/libkrb5support.so.0
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/ld- linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
0x007ef7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
[----- End lots of messages -----]

(gdb) bt
#0  0x007ef7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x008c60f3 in __write_nocancel () from /lib/tls/libc.so.6
#2  0x0064d734 in sock_write () from /s/openssl-0.9.8d/lib/libcrypto.so.0.9.8d
#3  0x00000008 in ?? ()
#4  0x09ede1a2 in ?? ()
#5  0x00000038 in ?? ()
#6  0x09ee50c0 in ?? ()
#7  0x006e06c8 in ?? () from /s/openssl- 0.9.8d/lib/libcrypto.so.0.9.8d
#8  0x09ed4148 in ?? ()
#9  0x00000000 in ?? ()

All the other processes are the same except for addresses in #4, #6, and #8, but they're all within a few MB of each other (they're probably asynchronous interrupts).

Let me know if you need more info.

Peter

On 5/22/07, Peter Koczan <pjkoczan@gmail.com> wrote:
The release is 8.2.4. I haven't been able to reproduce the condition yet, but I will send along stack traces as soon as I can. I have this strange feeling that it's only going to happen when I find a reason to make a restart-worthy config change.

Peter


On 5/21/07, Tom Lane < tgl@sss.pgh.pa.us> wrote:
"Peter Koczan" <pjkoczan@gmail.com> writes:
> [ lots of processes stuck in "notify interrupt" code ]

That's weird.  If it's still in that state, or if you can reproduce it,
could you attach to a few of those processes with gdb and get stack
traces?

Looking at the async.c code, an obvious candidate is that that routine
tries to take ExclusiveLock on pg_listener --- so if something had
managed to exit without releasing a lock on that table, hangups could be
expected.  But if that were the case, you'd think the process status
lines would include "waiting".  My guess is they're blocked on something
lower-level than a table lock, but without a stack trace it's hard to
guess what.

Which PG release is this exactly?

                        regards, tom lane


------=_Part_23756_13466382.1180630520238--