Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id F25EB2E002D for ; Sun, 24 Feb 2008 01:22:35 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 76919-07 for ; Sun, 24 Feb 2008 01:22:33 -0400 (AST) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from smtpoutm.mac.com (smtpoutm.mac.com [17.148.16.78]) by postgresql.org (Postfix) with ESMTP id 5ADF12E0033 for ; Sun, 24 Feb 2008 01:22:33 -0400 (AST) Received: from mac.com (asmtp008-s [10.150.69.71]) by smtpoutm.mac.com (Xserve/smtpout015/MantshX 4.0) with ESMTP id m1O5MW8p002751; Sat, 23 Feb 2008 21:22:32 -0800 (PST) Received: from [10.0.1.253] (75-144-26-241-sfba-ca.hfc.comcastbusiness.net [75.144.26.241] (may be forged)) (authenticated bits=0) by mac.com (Xserve/asmtp008/MantshX 4.0) with ESMTP id m1O5MTOd012868 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 23 Feb 2008 21:22:31 -0800 (PST) Mime-Version: 1.0 Message-Id: In-Reply-To: <6730.1203829093@sss.pgh.pa.us> References: <5576.1203795335@sss.pgh.pa.us> <6730.1203829093@sss.pgh.pa.us> Date: Sat, 23 Feb 2008 21:22:29 -0800 To: pgsql-performance@postgresql.org From: Joel Stevenson Subject: Re: LISTEN / NOTIFY performance in 8.3 Cc: Tom Lane Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200802/302 X-Sequence-Number: 28844 At 11:58 PM -0500 2/23/08, Tom Lane wrote: >Joel Stevenson writes: >>> That's strange, I would not have thought that listen/notify behavior >>> would change at all. How are you measuring this delay exactly? >>> Can you put together a self-contained test case? > >> Attached is a perl script that sort of simulates what's going on. > >Thanks for the script. It's not showing any particular problems here, >though. With log_min_duration_statement = 10, the only statements that >(slightly) exceed 10ms are the select count(*) from generate_series(1, >15000) ones. > >> Also of note, the iowait percentages on this quad core linux box jump >> to 30-40% while this test script is running, event though there's no >> table activity involved and the producer consumers pause for up to a >> second between iterations. > >This sounds a bit like pg_listener has gotten bloated. Try a "VACUUM >VERBOSE pg_listener" (as superuser) and see what it says. At the moment (server is inactive): pcdb=# VACUUM VERBOSE pg_listener; INFO: vacuuming "pg_catalog.pg_listener" INFO: "pg_listener": removed 1 row versions in 1 pages INFO: "pg_listener": found 1 removable, 21 nonremovable row versions in 28 pages DETAIL: 0 dead row versions cannot be removed yet. There were 2319 unused item pointers. 28 pages contain useful free space. 0 pages are entirely empty. CPU 0.00s/0.00u sec elapsed 0.00 sec. VACUUM running the test script and then the above command: pcdb=# VACUUM VERBOSE pg_listener; INFO: vacuuming "pg_catalog.pg_listener" INFO: "pg_listener": removed 693 row versions in 12 pages INFO: "pg_listener": found 693 removable, 21 nonremovable row versions in 28 pages DETAIL: 0 dead row versions cannot be removed yet. There were 2308 unused item pointers. 28 pages contain useful free space. 0 pages are entirely empty. CPU 0.00s/0.00u sec elapsed 0.00 sec. VACUUM Numerous notifications took 1000ms or so to complete in the test script execution between those two vacuum runs. >If that is the problem then the next question is why it got so much more >bloated than you were used to --- something wrong with vacuuming >procedures, perhaps? I have autovacuum on and using default settings. I have an explicit vacuum routine that runs nightly over the whole DB. -Joel