From: Mark Kirkwood <markir@coretech.co.nz>
To: Thomas F.O'Connell <tfo@sitening.com>
Cc: PgSQL - Performance <pgsql-performance@postgresql.org>
Subject: Re: Triggers During COPY
Date: Fri, 28 Jan 2005 18:55:37 +1300
Message-ID: <41F9D3D9.40909@coretech.co.nz> (raw)
In-Reply-To: <f2a4e35d83b33da9b5d007d276df484a@sitening.com>
References: <f2a4e35d83b33da9b5d007d276df484a@sitening.com>
Thomas F.O'Connell wrote:
>
> The problem comes in importing new data into the tables for which the
> counts are maintained. The current import process does some
> preprocessing and then does a COPY from the filesystem to one of the
> tables on which counts are maintained. This means that for each row
> being inserted by COPY, a trigger is fired. This didn't seem like a big
> deal to me until testing began on realistic data sets.
>
> For a 5,000-record import, preprocessing plus the COPY took about 5
> minutes. Once the triggers used for maintaining the counts were added,
> this grew to 25 minutes. While I knew there would be a slowdown per row
> affected, I expected something closer to 2x than to 5x.
> rformance out of this scenario?
>
Have been seeing similar behavior whilst testing sample code for the 8.0
docs (summary table plpgsql trigger example).
I think the nub of the problem is dead tuples bloat in the summary /
count table, so each additional triggered update becomes more and more
expensive as time goes on. I suspect the performance decrease is
exponential with the no of rows to be processed.
> Would it be absurd to drop the triggers during import and recreate them
> afterward and update the counts in a summary update based on information
> from the import process?
>
>
That's the conclusion I came to :-)
regards
Mark
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: markir@coretech.co.nz, tfo@sitening.com
Subject: Re: Triggers During COPY
In-Reply-To: <41F9D3D9.40909@coretech.co.nz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox