Received: from localhost (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id 0229D6321D9 for ; Tue, 13 Jan 2009 18:42:51 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 36747-07 for ; Tue, 13 Jan 2009 18:42:47 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.int.journyx.com (mail.journyx.com [66.219.41.227]) by mail.postgresql.org (Postfix) with ESMTP id EA7DB6321CB for ; Tue, 13 Jan 2009 18:42:47 -0400 (AST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.int.journyx.com (Postfix) with ESMTP id C53B37E40B3 for ; Tue, 13 Jan 2009 16:42:19 -0600 (CST) X-Virus-Scanned: amavisd-new at zimbra.int.journyx.com Received: from mail.int.journyx.com ([127.0.0.1]) by localhost (mail.int.journyx.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id baUsP6KRGi+E for ; Tue, 13 Jan 2009 16:41:47 -0600 (CST) Received: from whitneys (dhcp-100-188.int.journyx.com [192.168.100.188]) by mail.int.journyx.com (Postfix) with ESMTP id BDF247E46BA for ; Tue, 13 Jan 2009 16:34:07 -0600 (CST) From: "Scott Whitney" To: Subject: Re: Strange deadlock error last night Date: Tue, 13 Jan 2009 16:35:12 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 In-Reply-To: Thread-Index: Acl1zqd1e1OpYo9yRkaV0eOUvRqWBQAAEFvw X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 Message-Id: <20090113223407.BDF247E46BA@mail.int.journyx.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200901/39 X-Sequence-Number: 31086 Thanks for all the information, guys. I think Tom was right. Our application was doing a couple of full vacs at the same time. It's weird that we didn't run into this in the past. You're all absolutely right about the upgrading, but in our environment, it's not 2-3 minutes. It's 2-3 weeks. I've got to fully vet the app on the platform internally with full test plans, etc, even for the most minor upgrades; corp policy. Right now, my effort is in going to the latest stable branch. Moving forward, I will use these notes to get the company to revisit the minor upgrade policy, though. After all, when I _do_ get hit by one of those bugs, I _will_ be asked why we weren't upgraded. *sigh* -----Original Message----- From: Scott Marlowe [mailto:scott.marlowe@gmail.com] Sent: Tuesday, January 13, 2009 4:16 PM To: Scott Whitney Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Strange deadlock error last night On Tue, Jan 13, 2009 at 10:37 AM, Scott Whitney wrote: > It ended up locking up about 250 customer databases until I restarted the > postmaster. This is version 8.1.4. Upgrading right now (even to a minor rev) > is not really an option. This box has been up and running for 306 days. This > postgres level has been installed for..err...well...at least Aug 9, 2006, > based on some dates in the directories. You need to ask yourself how much downtime you can afford. The 2 or 3 minutes every few months to go from 8.1.x to 8.1.x+1, or the half a day of downtime when some horrendous bug takes down the whole site because you didn't update it. Seriously, that unfozen template0 bug that Alvarro mentioned is one of those kinds of bugs. Nothing like your db going down in the middle of the day with an error message that it's going down to prevent txid wraparound induced loss, please run vacuum on all your databases in single user mode. If you can't find set aside a minute or two at 0200 hrs, then don't be surprised when you get one of those failures.