X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 39DC53A6120 for ; Fri, 28 Jan 2005 05:51:45 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 03927-07 for ; Fri, 28 Jan 2005 05:51:35 +0000 (GMT) Received: from mail.coretech.co.nz (coretech.co.nz [202.36.204.41]) by svr1.postgresql.org (Postfix) with ESMTP id 8E5613A61D6 for ; Fri, 28 Jan 2005 05:51:34 +0000 (GMT) Received: (qmail 16735 invoked from network); 28 Jan 2005 05:51:28 -0000 Received: from 218-101-14-73.paradise.net.nz (HELO ?192.168.1.11?) (218.101.14.73) by 0 with SMTP; 28 Jan 2005 05:51:28 -0000 Message-ID: <41F9D3D9.40909@coretech.co.nz> Date: Fri, 28 Jan 2005 18:55:37 +1300 From: Mark Kirkwood User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20041016 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Thomas F.O'Connell" Cc: PgSQL - Performance Subject: Re: Triggers During COPY References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200501/565 X-Sequence-Number: 10255 Thomas F.O'Connell wrote: > > The problem comes in importing new data into the tables for which the > counts are maintained. The current import process does some > preprocessing and then does a COPY from the filesystem to one of the > tables on which counts are maintained. This means that for each row > being inserted by COPY, a trigger is fired. This didn't seem like a big > deal to me until testing began on realistic data sets. > > For a 5,000-record import, preprocessing plus the COPY took about 5 > minutes. Once the triggers used for maintaining the counts were added, > this grew to 25 minutes. While I knew there would be a slowdown per row > affected, I expected something closer to 2x than to 5x. > rformance out of this scenario? > Have been seeing similar behavior whilst testing sample code for the 8.0 docs (summary table plpgsql trigger example). I think the nub of the problem is dead tuples bloat in the summary / count table, so each additional triggered update becomes more and more expensive as time goes on. I suspect the performance decrease is exponential with the no of rows to be processed. > Would it be absurd to drop the triggers during import and recreate them > afterward and update the counts in a summary update based on information > from the import process? > > That's the conclusion I came to :-) regards Mark