pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Kynn Jones <kynnjo@gmail.com>
To: pgsql-performance@postgresql.org
Subject: How to "unique-ify" HUGE table?
Date: Tue, 23 Dec 2008 12:25:48 -0500
Message-ID: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com> (raw)

Hi everyone!
I have a very large 2-column table (about 500M records) from which I want to
remove duplicate records.

I have tried many approaches, but they all take forever.

The table's definition consists of two short TEXT columns.  It is a
temporary table generated from a query:

CREATE TEMP TABLE huge_table AS SELECT x, y FROM ... ;

Initially I tried

CREATE TEMP TABLE huge_table AS SELECT DISTINCT x, y FROM ... ;

but after waiting for nearly an hour I aborted the query, and repeated it
after getting rid of the DISTINCT clause.

Everything takes forever with this monster!  It's uncanny.  Even printing it
out to a file takes forever, let alone creating an index for it.

Any words of wisdom on how to speed this up would be appreciated.

TIA!

Kynn

view thread (7+ messages)  latest in thread

Message-ID: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com>
Permalink:  ../c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com/
Also on:    postgresql.org/message-id/c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: kynnjo@gmail.com
  Subject: Re: How to "unique-ify" HUGE table?
  In-Reply-To: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com>
  References: <c2350ba40812230925jb50fed7h3dc58cc311888c7f@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox