From imartinez@vectorsf.com Mon Jun 27 11:41:06 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 19D16B5DBC4 for ; Mon, 27 Jun 2011 08:41:06 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 87292-09 for ; Mon, 27 Jun 2011 11:41:00 +0000 (UTC) X-Greylist: delayed 00:09:39.47381 by SQLgrey-1.7.6 Received: from hermes.vectorsf.com (firewall.vectorsf.com [62.82.80.62]) by mail.postgresql.org (Postfix) with ESMTP id 8F644B5D94C for ; Mon, 27 Jun 2011 08:40:59 -0300 (ADT) Received: from localhost (localhost.localdomain [127.0.0.1]) by hermes.vectorsf.com (Postfix) with ESMTP id 8E4E411C0149 for ; Mon, 27 Jun 2011 13:31:17 +0200 (CEST) X-Virus-Scanned: amavisd-new at hermes.vectorsf.com Received: from hermes.vectorsf.com ([127.0.0.1]) by localhost (hermes.vectorsf.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tNXUj5dVZxuU for ; Mon, 27 Jun 2011 13:31:16 +0200 (CEST) Received: from [10.70.4.200] (unknown [10.70.4.200]) (Authenticated sender: imartinez) by hermes.vectorsf.com (Postfix) with ESMTPSA id 95E5711C0144 for ; Mon, 27 Jun 2011 13:31:16 +0200 (CEST) Subject: Urgent! Log full with inexistent PID processes From: =?ISO-8859-1?Q?I=F1igo?= Martinez Lasala To: pgsql-admin@postgresql.org Content-Type: multipart/alternative; boundary="=-h4IOjlLN81xzdfqIOMje" Date: Mon, 27 Jun 2011 13:31:17 +0200 Message-ID: <1309174277.5158.30.camel@deimos> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/142 X-Sequence-Number: 37768 --=-h4IOjlLN81xzdfqIOMje Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Hi everybody. We have a postgres 8.4.5-2 database under Debian Lenny 64bit. This database has been working without any problem, but this afternoon server has got out of space due to a log dump. Log file gets full with this entry until it reaches several gigabytes: 2011-06-27 13:25:07 CEST LOG: PID 21822 in cancel request did not match any process 2011-06-27 13:25:07 CEST LOG: PID 21821 in cancel request did not match any process 2011-06-27 13:25:07 CEST LOG: PID 21820 in cancel request did not match any process 2011-06-27 13:25:07 CEST LOG: PID 21816 in cancel request did not match any process 2011-06-27 13:25:07 CEST LOG: PID 21802 in cancel request did not match any process Of course, we have restarted postmaster and rebooted machine but problem remains. Any clue about this? pg_hba only allows connection from localhost and, of course, there are no processes with these PIDs in the server... Thanks in advance. --=-h4IOjlLN81xzdfqIOMje Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit Hi everybody.

We have a postgres 8.4.5-2 database under Debian Lenny 64bit.
This database has been working without any problem, but this afternoon server has got out of space due to a log dump.

Log file gets full with this entry until it reaches several gigabytes:

2011-06-27 13:25:07 CEST LOG:  PID 21822 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21821 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21820 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21816 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21802 in cancel request did not match any process


Of course, we have restarted postmaster and rebooted machine but problem remains.

Any clue about this?

pg_hba only allows connection from localhost and, of course, there are no processes with these PIDs in the server...

Thanks in advance.  --=-h4IOjlLN81xzdfqIOMje-- From adarsh.sharma@orkash.com Mon Jun 27 12:25:57 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 1F47CB5DBCB for ; Mon, 27 Jun 2011 09:25:57 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 75804-01 for ; Mon, 27 Jun 2011 12:25:50 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from smtp162.dfw.emailsrvr.com (smtp162.dfw.emailsrvr.com [67.192.241.162]) by mail.postgresql.org (Postfix) with ESMTP id C9335B5D94C for ; Mon, 27 Jun 2011 09:25:50 -0300 (ADT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp6.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id 43CB22701D1; Mon, 27 Jun 2011 08:25:50 -0400 (EDT) X-Virus-Scanned: OK Received: by smtp6.relay.dfw1a.emailsrvr.com (Authenticated sender: adarsh.sharma-AT-orkash.com) with ESMTPSA id 1ABE22701E5; Mon, 27 Jun 2011 08:25:48 -0400 (EDT) Message-ID: <4E0876C8.7050200@orkash.com> Date: Mon, 27 Jun 2011 17:55:44 +0530 From: Adarsh Sharma User-Agent: Thunderbird 2.0.0.22 (X11/20090625) MIME-Version: 1.0 To: =?UTF-8?B?ScOxaWdvIE1hcnRpbmV6IExhc2FsYQ==?= CC: pgsql-admin@postgresql.org Subject: Re: Urgent! Log full with inexistent PID processes References: <1309174277.5158.30.camel@deimos> In-Reply-To: <1309174277.5158.30.camel@deimos> Content-Type: multipart/alternative; boundary="------------030009020105010509000200" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201106/143 X-Sequence-Number: 37769 This is a multi-part message in MIME format. --------------030009020105010509000200 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Iñigo Martinez Lasala wrote: > Hi everybody. > > We have a postgres 8.4.5-2 database under Debian Lenny 64bit. > This database has been working without any problem, but this afternoon > server has got out of space due to a log dump. > > Log file gets full with this entry until it reaches several gigabytes: > > 2011-06-27 13:25:07 CEST LOG: PID 21822 in cancel request did not > match any process > 2011-06-27 13:25:07 CEST LOG: PID 21821 in cancel request did not > match any process > 2011-06-27 13:25:07 CEST LOG: PID 21820 in cancel request did not > match any process > 2011-06-27 13:25:07 CEST LOG: PID 21816 in cancel request did not > match any process > 2011-06-27 13:25:07 CEST LOG: PID 21802 in cancel request did not > match any process > > First check the space occupied by pg_log directory. Just delete the previous logs that are 1 month older. rm -rf pg_log****** > Of course, we have restarted postmaster and rebooted machine but > problem remains. > > Any clue about this? > > pg_hba only allows connection from localhost and, of course, there are > no processes with these PIDs in the server... > > Thanks in advance. --------------030009020105010509000200 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Iñigo Martinez Lasala wrote:
Hi everybody.

We have a postgres 8.4.5-2 database under Debian Lenny 64bit.
This database has been working without any problem, but this afternoon server has got out of space due to a log dump.

Log file gets full with this entry until it reaches several gigabytes:

2011-06-27 13:25:07 CEST LOG:  PID 21822 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21821 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21820 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21816 in cancel request did not match any process
2011-06-27 13:25:07 CEST LOG:  PID 21802 in cancel request did not match any process


First check the space occupied by pg_log directory.
Just delete the previous logs that are 1 month older.
rm -rf pg_log******


Of course, we have restarted postmaster and rebooted machine but problem remains.

Any clue about this?

pg_hba only allows connection from localhost and, of course, there are no processes with these PIDs in the server...

Thanks in advance. 

--------------030009020105010509000200-- From fweimer@bfk.de Mon Jun 27 13:40:59 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 0A374B5DBFF for ; Mon, 27 Jun 2011 10:40:59 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 91726-06 for ; Mon, 27 Jun 2011 13:40:52 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mx01.bfk.de (mx01.bfk.de [193.227.124.2]) by mail.postgresql.org (Postfix) with ESMTP id 0398FB5DBFA for ; Mon, 27 Jun 2011 10:40:51 -0300 (ADT) Received: from mx00.int.bfk.de ([10.119.110.2]) by mx01.bfk.de with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) id 1QbC3V-0001LR-Ns; Mon, 27 Jun 2011 13:40:49 +0000 Received: by bfk.de with local id 1QbC3V-0002Er-H8; Mon, 27 Jun 2011 13:40:49 +0000 From: Florian Weimer To: =?iso-8859-1?Q?I=F1igo?= Martinez Lasala Cc: pgsql-admin@postgresql.org Subject: Re: Urgent! Log full with inexistent PID processes References: <1309174277.5158.30.camel@deimos> Date: Mon, 27 Jun 2011 13:40:49 +0000 In-Reply-To: <1309174277.5158.30.camel@deimos> (=?iso-8859-1?Q?=22I=F1igo?= Martinez Lasala"'s message of "Mon, 27 Jun 2011 13:31:17 +0200") Message-ID: <82tybbs87y.fsf@mid.bfk.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/144 X-Sequence-Number: 37770 * I=F1igo Martinez Lasala: > Any clue about this? There is a misbehaving client which tries to issue broken cancel requests. You need to find it. > pg_hba only allows connection from localhost and, of course, there are > no processes with these PIDs in the server...=20 Do you use TCP or UNIX domain sockets for the database connections? --=20 Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstra=DFe 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 From imartinez@vectorsf.com Tue Jun 28 09:20:30 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 03595B5DBD8 for ; Tue, 28 Jun 2011 06:20:30 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 64128-01 for ; Tue, 28 Jun 2011 09:20:23 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from hermes.vectorsf.com (firewall.vectorsf.com [62.82.80.62]) by mail.postgresql.org (Postfix) with ESMTP id 44C46B5DBE0 for ; Tue, 28 Jun 2011 06:20:22 -0300 (ADT) Received: from localhost (localhost.localdomain [127.0.0.1]) by hermes.vectorsf.com (Postfix) with ESMTP id 18C2411C0165; Tue, 28 Jun 2011 11:20:19 +0200 (CEST) X-Virus-Scanned: amavisd-new at hermes.vectorsf.com Received: from hermes.vectorsf.com ([127.0.0.1]) by localhost (hermes.vectorsf.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EeZox7MKrHps; Tue, 28 Jun 2011 11:20:17 +0200 (CEST) Received: from [10.70.4.200] (unknown [10.70.4.200]) (Authenticated sender: imartinez) by hermes.vectorsf.com (Postfix) with ESMTPSA id DDC7411C0046; Tue, 28 Jun 2011 11:20:17 +0200 (CEST) Subject: Re: Urgent! Log full with inexistent PID processes From: =?ISO-8859-1?Q?I=F1igo?= Martinez Lasala To: Florian Weimer Cc: pgsql-admin@postgresql.org In-Reply-To: <82tybbs87y.fsf@mid.bfk.de> References: <1309174277.5158.30.camel@deimos> <82tybbs87y.fsf@mid.bfk.de> Content-Type: multipart/alternative; boundary="=-3zYsE8OzyeVhJ6/oleFp" Date: Tue, 28 Jun 2011 11:20:18 +0200 Message-ID: <1309252818.5158.84.camel@deimos> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/146 X-Sequence-Number: 37772 --=-3zYsE8OzyeVhJ6/oleFp Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Hi Florian. There is are no connections to DB. Only app using database is a web app that has been stopped also. -----Original Message----- From: Florian Weimer To: Iñigo Martinez Lasala Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Urgent! Log full with inexistent PID processes Date: Mon, 27 Jun 2011 13:40:49 +0000 * Iñigo Martinez Lasala: > Any clue about this? There is a misbehaving client which tries to issue broken cancel requests. You need to find it. > pg_hba only allows connection from localhost and, of course, there are > no processes with these PIDs in the server... Do you use TCP or UNIX domain sockets for the database connections? --=-3zYsE8OzyeVhJ6/oleFp Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit Hi Florian.

There is are no connections to DB. Only app using database is a web app that has been stopped also.

-----Original Message-----
From: Florian Weimer <fweimer@bfk.de>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Urgent! Log full with inexistent PID processes
Date: Mon, 27 Jun 2011 13:40:49 +0000

* Iñigo Martinez Lasala:

> Any clue about this?

There is a misbehaving client which tries to issue broken cancel
requests.  You need to find it.

> pg_hba only allows connection from localhost and, of course, there are
> no processes with these PIDs in the server... 

Do you use TCP or UNIX domain sockets for the database connections?

--=-3zYsE8OzyeVhJ6/oleFp-- From fweimer@bfk.de Tue Jun 28 09:25:34 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 4F116B5DBCB for ; Tue, 28 Jun 2011 06:25:34 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 61399-04 for ; Tue, 28 Jun 2011 09:25:28 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mx01.bfk.de (mx01.bfk.de [193.227.124.2]) by mail.postgresql.org (Postfix) with ESMTP id BBBDDB5DBC5 for ; Tue, 28 Jun 2011 06:25:27 -0300 (ADT) Received: from mx00.int.bfk.de ([10.119.110.2]) by mx01.bfk.de with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) id 1QbUXu-000528-H7; Tue, 28 Jun 2011 09:25:26 +0000 Received: by bfk.de with local id 1QbUXu-0000xc-D8; Tue, 28 Jun 2011 09:25:26 +0000 From: Florian Weimer To: =?iso-8859-1?Q?I=F1igo?= Martinez Lasala Cc: pgsql-admin@postgresql.org Subject: Re: Urgent! Log full with inexistent PID processes References: <1309174277.5158.30.camel@deimos> <82tybbs87y.fsf@mid.bfk.de> <1309252818.5158.84.camel@deimos> Date: Tue, 28 Jun 2011 09:25:26 +0000 In-Reply-To: <1309252818.5158.84.camel@deimos> (=?iso-8859-1?Q?=22I=F1igo?= Martinez Lasala"'s message of "Tue, 28 Jun 2011 11:20:18 +0200") Message-ID: <8239iumho9.fsf@mid.bfk.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-0.51 tagged_above=-10 required=5 tests=BAYES_05=-0.5, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/147 X-Sequence-Number: 37773 * I=F1igo Martinez Lasala: > There is are no connections to DB. Only app using database is a web app > that has been stopped also. Uh-oh. Can you "strace -f" the postmaster? Surely something is connecting to it. --=20 Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstra=DFe 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 From imartinez@vectorsf.com Tue Jun 28 09:38:59 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 02179B5DBC5 for ; Tue, 28 Jun 2011 06:38:59 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 72403-04 for ; Tue, 28 Jun 2011 09:38:53 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from hermes.vectorsf.com (firewall.vectorsf.com [62.82.80.62]) by mail.postgresql.org (Postfix) with ESMTP id A7FA5B5D803 for ; Tue, 28 Jun 2011 06:38:52 -0300 (ADT) Received: from localhost (localhost.localdomain [127.0.0.1]) by hermes.vectorsf.com (Postfix) with ESMTP id 705C811C0161; Tue, 28 Jun 2011 11:38:50 +0200 (CEST) X-Virus-Scanned: amavisd-new at hermes.vectorsf.com Received: from hermes.vectorsf.com ([127.0.0.1]) by localhost (hermes.vectorsf.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V3fNM4dSPa8D; Tue, 28 Jun 2011 11:38:46 +0200 (CEST) Received: from [10.70.4.200] (unknown [10.70.4.200]) (Authenticated sender: imartinez) by hermes.vectorsf.com (Postfix) with ESMTPSA id 75D1611C00DE; Tue, 28 Jun 2011 11:38:46 +0200 (CEST) Subject: Re: Urgent! Log full with inexistent PID processes From: =?ISO-8859-1?Q?I=F1igo?= Martinez Lasala To: Florian Weimer Cc: pgsql-admin@postgresql.org In-Reply-To: <8239iumho9.fsf@mid.bfk.de> References: <1309174277.5158.30.camel@deimos> <82tybbs87y.fsf@mid.bfk.de> <1309252818.5158.84.camel@deimos> <8239iumho9.fsf@mid.bfk.de> Content-Type: multipart/alternative; boundary="=-eQTzVFVp7SUS8/E8SSlY" Date: Tue, 28 Jun 2011 11:38:47 +0200 Message-ID: <1309253927.5158.89.camel@deimos> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/148 X-Sequence-Number: 37774 --=-eQTzVFVp7SUS8/E8SSlY Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit You are my man, bud. :) A remote server was launching a process against this database. We had limited access via pg_hba.conf (or at least we thought access was limited). Since a database admin was modifiying config in paralell with me, we have a mess with config and so access restriction was not applying. Thank you very mucho. -----Original Message----- From: Florian Weimer To: Iñigo Martinez Lasala Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Urgent! Log full with inexistent PID processes Date: Tue, 28 Jun 2011 09:25:26 +0000 * Iñigo Martinez Lasala: > There is are no connections to DB. Only app using database is a web app > that has been stopped also. Uh-oh. Can you "strace -f" the postmaster? Surely something is connecting to it. --=-eQTzVFVp7SUS8/E8SSlY Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit You are my man, bud. :)

A remote server was launching a process against this database. We had limited access via pg_hba.conf (or at least we thought access was limited). Since a database admin was modifiying config in paralell with me, we have a mess with config and so access restriction was not applying.

Thank you very mucho.

-----Original Message-----
From: Florian Weimer <fweimer@bfk.de>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Urgent! Log full with inexistent PID processes
Date: Tue, 28 Jun 2011 09:25:26 +0000

* Iñigo Martinez Lasala:

> There is are no connections to DB. Only app using database is a web app
> that has been stopped also.

Uh-oh.  Can you "strace -f" the postmaster?  Surely something is
connecting to it.


--=-eQTzVFVp7SUS8/E8SSlY--