From: Kenneth Marshall <ktm@rice.edu>
To: Marc Mamin <M.Mamin@intershop.de>
Cc: pgsql-performance@postgresql.org
Subject: Re: CREATE INDEX as bottleneck
Date: Thu, 11 Nov 2010 07:58:00 -0600
Message-ID: <20101111135800.GC14016@aart.is.rice.edu> (raw)
In-Reply-To: <C4DAC901169B624F933534A26ED7DF31034BB8D1@JENMAIL01.ad.intershop.net>
References: <C4DAC901169B624F933534A26ED7DF31034BB8D1@JENMAIL01.ad.intershop.net>
On Thu, Nov 11, 2010 at 02:41:12PM +0100, Marc Mamin wrote:
> Hello,
>
> in the last years, we have successfully manage to cope with our data
> growth
> using partitioning and splitting large aggregation tasks on multiple
> threads.
> The partitioning is done logically by our applicationn server, thus
> avoiding trigger overhead.
>
> There are a few places in our data flow where we have to wait for index
> creation before being able to distribute the process on multiple threads
> again.
>
> With the expected growth, create index will probably become a severe
> bottleneck for us.
>
> Is there any chance to see major improvement on it in a middle future ?
> I guess the question is naive, but why can't posgres use multiple
> threads for large sort operation ?
>
>
> best regards,
>
> Marc Mamin
>
There has been a recent discussion on the hackers mailing list on
using the infrastructure that is already in place to lauch autovacuum
processes to launch other helper processes. Something like this could
be used to offload the sort process to a much more parallelize version
that could take advantage of multiple I/O streams and CPU cores. Many
things are possible given the appropriate resources: funding, coding
and development cycles...
Regards,
Ken
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: ktm@rice.edu, M.Mamin@intershop.de
Subject: Re: CREATE INDEX as bottleneck
In-Reply-To: <20101111135800.GC14016@aart.is.rice.edu>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox