From: runner <runner@winning.com>
To: pgsql-performance@postgresql.org
Subject: Re: Tuning massive UPDATES and GROUP BY's?
Date: Mon, 14 Mar 2011 11:54:39 -0400
Message-ID: <8CDB0773FE92165-27F8-BD4@web-mmc-m04.sysops.aol.com> (raw)
In-Reply-To: <AANLkTikqxB3EmnvgxYEGPwOg50r0K1JD=u651puWOzqW@mail.gmail.com>
References: <loom.20110310T163020-437@post.gmane.org>
<AANLkTiks9aTwVyN-JW1Ep2cVoJabBF_fZfkk_43GRcNv@mail.gmail.com>
<loom.20110311T195504-383@post.gmane.org>
<AANLkTi=dGuvB_EuU43CbFUeuaG57-WXvypw+A1WKHhZ8@mail.gmail.com>
<8CDAFB3EE5DFE5F-1AE0-21557@web-mmc-d02.sysops.aol.com>
<AANLkTimVFRO3FFGo3YXf3tSTVNdXRmLh8STnmJt9eTRr@mail.gmail.com>
<AANLkTikqxB3EmnvgxYEGPwOg50r0K1JD=u651puWOzqW@mail.gmail.com>
> Bulk data imports of this size I've done with minimal pain by simply
> breaking the raw data into chunks (10M records becomes 10 files of
> 1M records), on a separate spindle from the database, and performing
> multiple COPY commands but no more than 1 COPY per server core.
> I tested this a while back on a 4 core server and when I attempted 5
> COPY's at a time the time to complete went up almost 30%. I don't
> recall any benefit having fewer than 4 in this case but the server was
> only processing my data at the time. Indexes were on the target table
> however I dropped all constraints. The UNIX split command is handy
> for breaking the data up into individual files.
I'm not using COPY. My dump file is a bunch if INSERT INTO statements. I know it would be faster to use copy. If I can figure out how to do this in one hour I will try it. I did two mysqldumps, one with INSERT INTO and one as CSV to I can try COPY at a later time. I'm running five parallel psql processes to import the data which has been broken out by table.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: runner@winning.com
Subject: Re: Tuning massive UPDATES and GROUP BY's?
In-Reply-To: <8CDB0773FE92165-27F8-BD4@web-mmc-m04.sysops.aol.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox