Date: Thu, 11 Nov 2010 07:58:00 -0600
From: Kenneth Marshall <ktm@rice.edu>
To: Marc Mamin <M.Mamin@intershop.de>
Cc: pgsql-performance@postgresql.org
Subject: Re: CREATE INDEX as bottleneck
Message-ID: <20101111135800.GC14016@aart.is.rice.edu>
References: 
 <C4DAC901169B624F933534A26ED7DF31034BB8D1@JENMAIL01.ad.intershop.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <C4DAC901169B624F933534A26ED7DF31034BB8D1@JENMAIL01.ad.intershop.net>
User-Agent: Mutt/1.5.16 (2007-06-09)

On Thu, Nov 11, 2010 at 02:41:12PM +0100, Marc Mamin wrote:
> Hello,
> 
> in the last years, we have successfully manage to cope with our data
> growth 
> using partitioning and splitting large aggregation tasks on multiple
> threads.
> The partitioning is done logically by our applicationn server, thus
> avoiding trigger overhead.
> 
> There are a few places in our data flow where we have to wait for index
> creation before being able to distribute the process on multiple threads
> again.
> 
> With the expected growth, create index will probably become a severe
> bottleneck for us.
> 
> Is there any chance to see major improvement on it in a middle future ?
> I guess the question is naive, but why can't posgres use multiple
> threads for large sort operation ?
> 
> 
> best regards,
> 
> Marc Mamin
> 

There has been a recent discussion on the hackers mailing list on
using the infrastructure that is already in place to lauch autovacuum
processes to launch other helper processes. Something like this could
be used to offload the sort process to a much more parallelize version
that could take advantage of multiple I/O streams and CPU cores. Many
things are possible given the appropriate resources: funding, coding
and development cycles...

Regards,
Ken