pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Mark Kirkwood <markir@coretech.co.nz>
To: Thomas F.O'Connell <tfo@sitening.com>
Cc: PgSQL - Performance <pgsql-performance@postgresql.org>
Subject: Re: Triggers During COPY
Date: Fri, 28 Jan 2005 18:55:37 +1300
Message-ID: <41F9D3D9.40909@coretech.co.nz> (raw)
In-Reply-To: <f2a4e35d83b33da9b5d007d276df484a@sitening.com>
References: <f2a4e35d83b33da9b5d007d276df484a@sitening.com>

Thomas F.O'Connell wrote:
> 
> The problem comes in importing new data into the tables for which the 
> counts are maintained. The current import process does some 
> preprocessing and then does a COPY from the filesystem to one of the 
> tables on which counts are maintained. This means that for each row 
> being inserted by COPY, a trigger is fired. This didn't seem like a big 
> deal to me until testing began on realistic data sets.
> 
> For a 5,000-record import, preprocessing plus the COPY took about 5 
> minutes. Once the triggers used for maintaining the counts were added, 
> this grew to 25 minutes. While I knew there would be a slowdown per row 
> affected, I expected something closer to 2x than to 5x.
> rformance out of this scenario?
> 
Have been seeing similar behavior whilst testing sample code for the 8.0
docs (summary table plpgsql trigger example).

I think the nub of the problem is dead tuples bloat in the summary /
count table, so each additional triggered update becomes more and more
expensive as time goes on. I suspect the performance decrease is
exponential with the no of rows to be processed.


> Would it be absurd to drop the triggers during import and recreate them 
> afterward and update the counts in a summary update based on information 
> from the import process?
> 
> 
That's the conclusion I came to :-)

regards

Mark




view thread (6+ messages)  latest in thread

Message-ID: <41F9D3D9.40909@coretech.co.nz>
Permalink:  ../41F9D3D9.40909@coretech.co.nz/
Also on:    postgresql.org/message-id/41F9D3D9.40909@coretech.co.nz

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: markir@coretech.co.nz, tfo@sitening.com
  Subject: Re: Triggers During COPY
  In-Reply-To: <41F9D3D9.40909@coretech.co.nz>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox