From: Scott Marlowe <scott.marlowe@gmail.com>
To: Kynn Jones <kynnjo@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: How to "unique-ify" HUGE table?
Date: Tue, 23 Dec 2008 10:34:28 -0700
Message-ID: <dcc563d10812230934y613ec899sffe5483e87c93cfd@mail.gmail.com> (raw)
In-Reply-To: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com>
References: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com>
On Tue, Dec 23, 2008 at 10:25 AM, Kynn Jones <kynnjo@gmail.com> wrote:
> Hi everyone!
> I have a very large 2-column table (about 500M records) from which I want to
> remove duplicate records.
> I have tried many approaches, but they all take forever.
> The table's definition consists of two short TEXT columns. It is a
> temporary table generated from a query:
>
> CREATE TEMP TABLE huge_table AS SELECT x, y FROM ... ;
> Initially I tried
> CREATE TEMP TABLE huge_table AS SELECT DISTINCT x, y FROM ... ;
> but after waiting for nearly an hour I aborted the query, and repeated it
> after getting rid of the DISTINCT clause.
> Everything takes forever with this monster! It's uncanny. Even printing it
> out to a file takes forever, let alone creating an index for it.
> Any words of wisdom on how to speed this up would be appreciated.
Did you try cranking up work_mem to something that's a large
percentage (25 to 50%) of total memory?
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: scott.marlowe@gmail.com, kynnjo@gmail.com
Subject: Re: How to "unique-ify" HUGE table?
In-Reply-To: <dcc563d10812230934y613ec899sffe5483e87c93cfd@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox