From postgres@kiwi-fraser.net Fri Aug 25 11:33:19 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 89B5A9FB90F for ; Fri, 25 Aug 2006 08:33:19 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 17598-09 for ; Fri, 25 Aug 2006 08:33:14 -0300 (ADT) X-Greylist: delayed 00:19:11.773484 by SQLgrey- Received: from svr4.postgresql.org (svr4.postgresql.org [66.98.251.159]) by postgresql.org (Postfix) with ESMTP id 09D539FB916 for ; Fri, 25 Aug 2006 08:33:11 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.3 Received: from smtp-out2.blueyonder.co.uk (smtp-out2.blueyonder.co.uk [195.188.213.5]) by svr4.postgresql.org (Postfix) with ESMTP id DEC4E5AF893 for ; Fri, 25 Aug 2006 11:13:58 +0000 (GMT) Received: from [172.23.170.143] (helo=anti-virus02-10) by smtp-out2.blueyonder.co.uk with smtp (Exim 4.52) id 1GGZdM-0001b9-Fj for pgsql-admin@postgresql.org; Fri, 25 Aug 2006 12:13:56 +0100 Received: from [82.44.23.191] (helo=demolish1) by asmtp-out2.blueyonder.co.uk with smtp (Exim 4.52) id 1GGZdL-0002Jc-9c for pgsql-admin@postgresql.org; Fri, 25 Aug 2006 12:13:55 +0100 Message-ID: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> From: "Donald Fraser" To: "[ADMIN]" Subject: Strange message followed by server crash Date: Fri, 25 Aug 2006 12:13:54 +0100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0031_01C6C83F.EED85E50" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1437 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.497 tagged_above=0 required=5 tests=HTML_40_50, HTML_MESSAGE X-Spam-Level: X-Archive-Number: 200608/280 X-Sequence-Number: 22823 This is a multi-part message in MIME format. ------=_NextPart_000_0031_01C6C83F.EED85E50 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I'm trying to figure out what exactly caused PostgreSQL to crash. PostgreSQL 8.0.7 OS: Red Hat Linux EL 4.0 Log messages: <1Tmenteshvili 10709 2006-08-24 17:48:19 BST 0> ERROR: invalid message = format < 3670 2006-08-24 17:48:19 BST > LOG: server process (PID 10709) was = terminated by signal 11 < 3670 2006-08-24 17:48:19 BST > LOG: terminating any other active = server processes Looking into PostgreSQL code I note that the message "invalid message = format" is generated by function: pq_getmsgend which is called when = processing incoming messages from clients.=20 So am I correct in assuming that there was an error in the front = end/back end data sent to PostgreSQL? I asked the client above what they were doing when this event occurred = and the answer was logging out of the database. At the same time, an engineer, who thought nobody was in the building, = was working on the network and changing a network switch. Therefore can I assume this crash was caused by some form of corrupt / = incomplete data sent to PostgreSQL? Thanks in advance regards Donald Fraser ------=_NextPart_000_0031_01C6C83F.EED85E50 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I'm trying to figure out what exactly caused = PostgreSQL to=20 crash.
 
PostgreSQL 8.0.7
OS: Red Hat Linux EL 4.0
 
Log messages:
<1Tmenteshvili 10709 2006-08-24 17:48:19 BST = 0>=20 ERROR:  invalid message format
< 3670 2006-08-24 17:48:19 BST = >=20 LOG:  server process (PID 10709) was terminated by signal = 11
< 3670=20 2006-08-24 17:48:19 BST > LOG:  terminating any other active = server=20 processes
 
Looking into PostgreSQL code I note that the message "invalid = message=20 format" is generated by function: pq_getmsgend which is called when = processing incoming messages from clients.
So am I correct in assuming that there was an error in the front = end/back=20 end data sent to PostgreSQL?
 
I asked the client above what they were doing when this event = occurred and=20 the answer was logging out of the database.
 
At the same time, an engineer, who thought nobody was in = the=20 building, was working on the network and changing a network = switch.
Therefore can I assume this crash was caused by some form of = corrupt /=20 incomplete data sent to PostgreSQL?
 
Thanks in advance
regards
Donald Fraser
 
 
------=_NextPart_000_0031_01C6C83F.EED85E50-- From tgl@sss.pgh.pa.us Fri Aug 25 13:44:43 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 402E19FBA52 for ; Fri, 25 Aug 2006 10:44:43 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 43630-09 for ; Fri, 25 Aug 2006 10:44:38 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id 830C19FBA4D for ; Fri, 25 Aug 2006 10:44:38 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.6/8.13.6) with ESMTP id k7PDiMuo023161; Fri, 25 Aug 2006 09:44:22 -0400 (EDT) To: "Donald Fraser" cc: "[ADMIN]" Subject: Re: Strange message followed by server crash In-reply-to: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> References: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> Comments: In-reply-to "Donald Fraser" message dated "Fri, 25 Aug 2006 12:13:54 +0100" Date: Fri, 25 Aug 2006 09:44:22 -0400 Message-ID: <23160.1156513462@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.143 tagged_above=0 required=5 tests=AWL, SPF_HELO_PASS, SPF_PASS X-Spam-Level: X-Archive-Number: 200608/284 X-Sequence-Number: 22827 "Donald Fraser" writes: > Log messages: > <1Tmenteshvili 10709 2006-08-24 17:48:19 BST 0> ERROR: invalid message format > < 3670 2006-08-24 17:48:19 BST > LOG: server process (PID 10709) was terminated by signal 11 > At the same time, an engineer, who thought nobody was in the building, was working on the network and changing a network switch. Hm, the "invalid message format" is unsurprising if the network suffered a glitch and delivered bad data, but the backend crash is not good. And what do you make of the apparent junk on the log line? (What's your log_line_prefix exactly?) Given that this sounds like it won't be easy to reproduce, I'm hoping you have a core file left from that and can get a stack trace from it. That would give us some clue where to look. regards, tom lane From postgres@kiwi-fraser.net Fri Aug 25 14:46:28 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 391089FBAD6 for ; Fri, 25 Aug 2006 11:46:28 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 50883-05 for ; Fri, 25 Aug 2006 14:46:15 +0000 (UTC) X-Greylist: delayed 03:32:14.296204 by SQLgrey- Received: from smtp-out2.blueyonder.co.uk (smtp-out2.blueyonder.co.uk [195.188.213.5]) by postgresql.org (Postfix) with ESMTP id 65C739FBAD5 for ; Fri, 25 Aug 2006 11:46:15 -0300 (ADT) Received: from [172.23.170.143] (helo=anti-virus02-10) by smtp-out2.blueyonder.co.uk with smtp (Exim 4.52) id 1GGcwn-0001a2-OR for pgsql-admin@postgresql.org; Fri, 25 Aug 2006 15:46:13 +0100 Received: from [82.44.23.191] (helo=demolish1) by asmtp-out5.blueyonder.co.uk with smtp (Exim 4.52) id 1GGcwn-0001hY-0z for pgsql-admin@postgresql.org; Fri, 25 Aug 2006 15:46:13 +0100 Message-ID: <008c01c6c855$35f6cd20$7d64a8c0@demolish1> From: "Donald Fraser" To: "[ADMIN]" References: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> <23160.1156513462@sss.pgh.pa.us> Subject: Re: Strange message followed by server crash Date: Fri, 25 Aug 2006 15:46:11 +0100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0089_01C6C85D.97250FA0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1437 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.497 tagged_above=0 required=5 tests=HTML_40_50, HTML_MESSAGE X-Spam-Level: X-Archive-Number: 200608/287 X-Sequence-Number: 22830 This is a multi-part message in MIME format. ------=_NextPart_000_0089_01C6C85D.97250FA0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable >"Donald Fraser" writes: >> Log messages: >> <1Tmenteshvili 10709 2006-08-24 17:48:19 BST 0> ERROR: invalid = message format >> < 3670 2006-08-24 17:48:19 BST > LOG: server process (PID 10709) was = terminated by signal 11 >> At the same time, an engineer, who thought nobody was in the = building, was working on the network and changing a network switch. "Tom Lane" writes: >And what do you make of the apparent junk on the log line? (What's >your log_line_prefix exactly?) The log messages are good, log prefix is: <%u %p %t %x>=20 "1Tmenteshvili" is actually a user login name... don't ask. >Given that this sounds like it won't be easy to reproduce, I'm hoping >you have a core file left from that and can get a stack trace from it. >That would give us some clue where to look. Where would I look for a core file? There's nothing unusual in the data directory, nothing in the /tmp = directory? I had my email turned off yesterday and PostgreSQL restarted itself so I = didn't find out about the crash till this morning when I logged in. Everything seems to have be running smoothly since then... If there isn't a core file, how would I go about ensuring one is created = next time? (touch wood there wont be one). Regards Donald Fraser ------=_NextPart_000_0089_01C6C85D.97250FA0 Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
>"Donald Fraser" <postgres@kiwi-fraser.net>= =20 writes:
>> Log messages:
>> <1Tmenteshvili 10709 = 2006-08-24=20 17:48:19 BST 0> ERROR:  invalid message format
>> < = 3670=20 2006-08-24 17:48:19 BST > LOG:  server process (PID 10709) was=20 terminated by signal 11

>> At the same time, an engineer, = who=20 thought nobody was in the building, was working on the network and = changing a=20 network switch.
 
"Tom Lane" writes:
>And what do you = make of the=20 apparent junk on the log line?  (What's
>your log_line_prefix = exactly?)
The log messages are good, log prefix is: <%u %p = %t %x>=20
"1Tmenteshvili" is actually a user login name... don't = ask.

>Given=20 that this sounds like it won't be easy to reproduce, I'm = hoping
>you have=20 a core file left from that and can get a stack trace from = it.
>That would=20 give us some clue where to look.

Where would I = look for a=20 core file?
There's nothing unusual in the data directory, = nothing in the=20 /tmp directory?
 
I had my email turned off yesterday and PostgreSQL = restarted=20 itself so I didn't find out about the crash till this morning when I = logged=20 in.
Everything seems to have be running smoothly since=20 then...
If there isn't a core file, how would I go about = ensuring one=20 is created next time? (touch wood there wont be one).
 
Regards
Donald Fraser
 
------=_NextPart_000_0089_01C6C85D.97250FA0-- From tgl@sss.pgh.pa.us Fri Aug 25 15:05:40 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 0BA299FBB06 for ; Fri, 25 Aug 2006 12:05:40 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 73999-01 for ; Fri, 25 Aug 2006 15:05:31 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id 78A2F9FBB05 for ; Fri, 25 Aug 2006 12:05:31 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.6/8.13.6) with ESMTP id k7PF5Ohx024549; Fri, 25 Aug 2006 11:05:24 -0400 (EDT) To: "Donald Fraser" cc: "[ADMIN]" Subject: Re: Strange message followed by server crash In-reply-to: <008c01c6c855$35f6cd20$7d64a8c0@demolish1> References: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> <23160.1156513462@sss.pgh.pa.us> <008c01c6c855$35f6cd20$7d64a8c0@demolish1> Comments: In-reply-to "Donald Fraser" message dated "Fri, 25 Aug 2006 15:46:11 +0100" Date: Fri, 25 Aug 2006 11:05:24 -0400 Message-ID: <24548.1156518324@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200608/288 X-Sequence-Number: 22831 "Donald Fraser" writes: > "Tom Lane" writes: >> Given that this sounds like it won't be easy to reproduce, I'm hoping >> you have a core file left from that and can get a stack trace from it. > Where would I look for a core file? > There's nothing unusual in the data directory, nothing in the /tmp = > directory? Normally it would go into the $PGDATA/base/NNN subdirectory for the database the backend was connected to. A very few systems (OS X for instance) drop cores into a dedicated system-wide directory called /cores or some such. If you're not finding anything, it's probably because the postmaster was launched under "ulimit -c 0" or local equivalent spelling --- a lot of systems have a policy of launching daemons that way. > If there isn't a core file, how would I go about ensuring one is created = > next time? (touch wood there wont be one). Put "ulimit -c unlimited" into the script that launches the postmaster, and restart it. regards, tom lane From alvherre@commandprompt.com Fri Aug 25 15:24:52 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id C2BCC9FBB35 for ; Fri, 25 Aug 2006 12:24:52 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 77410-10 for ; Fri, 25 Aug 2006 12:24:47 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from lists.commandprompt.com (host-130.commandprompt.net [207.173.203.130]) by postgresql.org (Postfix) with ESMTP id 21E6B9FBB25 for ; Fri, 25 Aug 2006 12:24:46 -0300 (ADT) Received: from alvh.no-ip.org (201-221-200-187.bk11-dsl.surnet.cl [201.221.200.187]) (authenticated bits=0) by lists.commandprompt.com (8.13.7/8.13.6) with ESMTP id k7PFOT85030011; Fri, 25 Aug 2006 08:24:30 -0700 Received: by alvh.no-ip.org (Postfix, from userid 1000) id 717B1C2DC63; Fri, 25 Aug 2006 11:24:13 -0400 (CLT) Date: Fri, 25 Aug 2006 11:24:13 -0400 From: Alvaro Herrera To: Tom Lane Cc: Donald Fraser , "[ADMIN]" Subject: Re: Strange message followed by server crash Message-ID: <20060825152413.GG14622@alvh.no-ip.org> Mail-Followup-To: Tom Lane , Donald Fraser , "[ADMIN]" References: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> <23160.1156513462@sss.pgh.pa.us> <008c01c6c855$35f6cd20$7d64a8c0@demolish1> <24548.1156518324@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <24548.1156518324@sss.pgh.pa.us> User-Agent: Mutt/1.5.12-2006-07-14 X-Virus-Scanned: ClamAV version 0.88.3, clamav-milter version 0.88.3 on projects.commandprompt.com X-Virus-Status: Clean X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (lists.commandprompt.com [192.168.2.159]); Fri, 25 Aug 2006 08:24:30 -0700 (PDT) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.332 tagged_above=0 required=5 tests=AWL, FORGED_RCVD_HELO X-Spam-Level: X-Archive-Number: 200608/289 X-Sequence-Number: 22832 Tom Lane wrote: > "Donald Fraser" writes: > > "Tom Lane" writes: > >> Given that this sounds like it won't be easy to reproduce, I'm hoping > >> you have a core file left from that and can get a stack trace from it. > > > Where would I look for a core file? > > There's nothing unusual in the data directory, nothing in the /tmp = > > directory? > > Normally it would go into the $PGDATA/base/NNN subdirectory for the > database the backend was connected to. A very few systems (OS X for > instance) drop cores into a dedicated system-wide directory called > /cores or some such. Note that on recent versions of Linux you can modify both the file name and the directory where it's saved, by modifying /proc/sys/kernel/core_pattern (or the deprecated /proc/sys/kernel/core_uses_pid). See core(5) for details. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. From tgl@sss.pgh.pa.us Fri Sep 1 02:03:40 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 6C0959FB43F for ; Thu, 31 Aug 2006 23:03:40 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 74195-03-3 for ; Fri, 1 Sep 2006 02:02:59 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id 338D09FB475 for ; Thu, 31 Aug 2006 23:01:44 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.6/8.13.6) with ESMTP id k8121aO9027237; Thu, 31 Aug 2006 22:01:37 -0400 (EDT) To: "Donald Fraser" cc: "[ADMIN]" Subject: Re: Strange message followed by server crash In-reply-to: <008c01c6c855$35f6cd20$7d64a8c0@demolish1> References: <003401c6c837$8dc6cb90$7d64a8c0@demolish1> <23160.1156513462@sss.pgh.pa.us> <008c01c6c855$35f6cd20$7d64a8c0@demolish1> Comments: In-reply-to "Donald Fraser" message dated "Fri, 25 Aug 2006 15:46:11 +0100" Date: Thu, 31 Aug 2006 22:01:36 -0400 Message-ID: <27236.1157076096@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200608/371 X-Sequence-Number: 22914 "Donald Fraser" writes: > Log messages: > <1Tmenteshvili 10709 2006-08-24 17:48:19 BST 0> ERROR: invalid = > message format > < 3670 2006-08-24 17:48:19 BST > LOG: server process (PID 10709) was = > terminated by signal 11 > At the same time, an engineer, who thought nobody was in the = > building, was working on the network and changing a network switch. I tried to reproduce this by using gdb to force the "invalid message format" failure --- that is, deliberately changing one of the values pq_getmsgend compares. No luck, the backend issued the error and continued just as it should. My best guess now that the crash was not directly related to that error, but was caused by insufficiently robust processing of whatever garbage was received following that. However, with no clue what that garbage might have looked like, there's too many possible code paths to try to chase through with no leads. Anyone care to do some random-data stress testing, ie, connect to a backend and squirt random data at it to see if you can make it crash rather than just complain and disconnect? Be prepared to show a backend stack trace and the exact data sent if you succeed. regards, tom lane