X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id 8497611F66F4 for ; Fri, 7 Apr 2006 19:28:07 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 83955-09 for ; Fri, 7 Apr 2006 19:28:02 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from fon.nation-net.com (213-2-2-26.uk.vianw.net [213.2.2.26]) by postgresql.org (Postfix) with ESMTP id 6912611F66D9 for ; Fri, 7 Apr 2006 19:28:03 -0300 (ADT) Received: from fon.nation-net.com (localhost [127.0.0.1]) by fon.nation-net.com (Postfix) with ESMTP id BE9CC137692 for ; Fri, 7 Apr 2006 23:28:03 +0100 (BST) Received: from eddie.acentral.co.uk (80-192-144-33.cable.ubr04.pres.blueyonder.co.uk [80.192.144.33]) by fon.nation-net.com (Postfix) with ESMTP id 9E49C136CD9 for ; Fri, 7 Apr 2006 23:28:03 +0100 (BST) Received: from eddie.acentral.co.uk (eddie.acentral.co.uk [127.0.0.1]) by eddie.acentral.co.uk (Postfix) with ESMTP id 353B475FE5 for ; Fri, 7 Apr 2006 23:28:06 +0100 (BST) Received: from localhost.localdomain (unknown [10.0.0.29]) by eddie.acentral.co.uk (Postfix) with SMTP id 033F075F8D for ; Fri, 7 Apr 2006 23:28:05 +0100 (BST) Date: Fri, 7 Apr 2006 23:27:59 +0100 From: Gavin Hamill To: pgsql-performance@postgresql.org Subject: Re: pg 8.1.3, AIX, huge box, painfully slow. Message-Id: <20060407232759.45581362.gdh@laterooms.com> In-Reply-To: <17055.1144447009@sss.pgh.pa.us> References: <20060407185823.abe93017.gdh@laterooms.com> <1144436060.32269.13.camel@state.g2switchworks.com> <20060407205919.a091da26.gdh@laterooms.com> <15772.1144440362@sss.pgh.pa.us> <20060407223635.9c4a0b38.gdh@laterooms.com> <17055.1144447009@sss.pgh.pa.us> X-Mailer: Sylpheed version 2.1.1 (GTK+ 2.8.6; i486-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-AV-Checked: ClamAV using ClamSMTP X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.103 required=5 tests=[AWL=0.103] X-Spam-Score: 0.103 X-Spam-Level: X-Archive-Number: 200604/164 X-Sequence-Number: 18248 On Fri, 07 Apr 2006 17:56:49 -0400 Tom Lane wrote: > This is not good. Did the semop storms coincide with visible > slowdown? (I'd assume so, but you didn't actually say...) If I'd been able to tell, then I'd tell you =) I'll have another go... Yes, there's a definate correlation here.. I attached truss to the main postmaster.. $ truss -Ff -p 340344 2>&1 | grep semop here's a snippet 278774: __semop(15728650, 0x0FFFFFFFFFFF7E80, 1) = 0 155712: __semop(15728650, 0x0FFFFFFFFFFF5920, 1) = 0 278774: __semop(15728649, 0x0FFFFFFFFFFF6F10, 1) 114914: __semop(15728649, 0x0FFFFFFFFFFF6A40, 1) = 0 = 0 114914: __semop(15728650, 0x0FFFFFFFFFFF61E0, 1) 155712: __semop(15728650, 0x0FFFFFFFFFFF6850, 1) = 0 = 0 155712: __semop(15728650, 0x0FFFFFFFFFFF6890, 1) = 0 1 55712: __semop(15728650, 0x0FFFFFFFFFFF5920, 1) 278774: __semop(15728650, 0x0FFFFFFFFFFF6F10, 1) 155712: __semop(15728650, 0x0FFFFFFFFFFF6850, 1) = 0 = 0 278774: __semop(15728649, 0x0FFFFFFFFFFF7E40, 1) 114914: __semop(15728649, 0x0FFFFFFFFFFF6A80, 1) = 0 = 0 278774: __semop(15728650, 0x0FFFFFFFFFFF7E80, 1) And when I saw a flood of semop's for any particular PID, a second later in the 'topas' process list would show that PID at a 100% CPU ... Most intriguing :) Cheers, Gavin.