DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type;
	b=fwYxoaUznDmg8OqoGD5MCt+nAdvxLHz8Hw6ESQcrzUaLQQUkj0Seln7qPhHnojSg6I
	Fs28pI91CZI3pCeHOrxyB9ZfFCoGjFTbO6G2xBmvfFdUqmC/rmZENxTh+K8dgewtN8CN
	AAJsSf9+OoATOXJ6DkxVk642gpkvLfaHiQ+e8=
MIME-Version: 1.0
In-Reply-To: <hi41os$eiv$1@news.hub.org>
References: <hi41os$eiv$1@news.hub.org>
Date: Thu, 7 Jan 2010 00:49:30 -0700
Message-ID: <dcc563d11001062349y54bdbdcbm5d95fc5f08a332a8@mail.gmail.com>
Subject: Re: Massive table (500M rows) update nightmare
From: Scott Marlowe <scott.marlowe@gmail.com>
To: Carlo Stonebanks <stonec.register@sympatico.ca>
Cc: pgsql-performance@postgresql.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jan 7, 2010 at 12:17 AM, Carlo Stonebanks
<stonec.register@sympatico.ca> wrote:
> Our DB has an audit table which is 500M rows and growing. (FYI the objects
> being audited are grouped semantically, not individual field values).
>
> Recently we wanted to add a new feature and we altered the table to add a
> new column. We are backfilling this varchar(255) column by writing a TCL
> script to page through the rows (where every update is a UPDATE ... WHERE id
>>= x AND id < x+10 and a commit is performed after every 1000 updates
> statement, i.e. every 10000 rows.)
>
> We have 10 columns, six of which are indexed. Rough calculations suggest
> that this will take two to three weeks to complete on an 8-core CPU with
> more than enough memory.
>
> As a ballpark estimate - is this sort of performance for an 500M updates
> what one would expect of PG given the table structure (detailed below) or
> should I dig deeper to look for performance issues?

Got an explain analyze of the delete query?