Received: from localhost (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id 30CC16501D6 for ; Tue, 23 Dec 2008 13:25:53 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 20549-02 for ; Tue, 23 Dec 2008 13:25:50 -0400 (AST) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.233]) by mail.postgresql.org (Postfix) with ESMTP id E86966500F6 for ; Tue, 23 Dec 2008 13:25:49 -0400 (AST) Received: by rv-out-0506.google.com with SMTP id b25so3963760rvf.43 for ; Tue, 23 Dec 2008 09:25:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=P+G947SjW92E7E8aHT4NSIzWlRdYYz6RSeY+L1bfLjk=; b=kogaPiKsBo7MTHSqNeTqmN9zNHw3Izp1YBVHccJfz7H1nyj5Oa8WSJZMS1s2H5ki+j 9+zP6TyVDHSG5HuawX9gctWWjNjEDk+nWCgCRaj8HwtNqMAnKyALEcZohSgz35YruEby gRPNpLey744GndeiiFS0ljI4V5LeXfUhcCzIc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=fqpdw93jckajGkaIuAy1qQq7/VKyZix79DOKaWHmMKhMvLO2VU262ZxvzNKShqOP4E C9t8tU6LzE+YtiaM7ohQrBOF/ugfpAa+DA5gi1dsLCJ9+LquS3eyRCmJRpY+Jco6ZyvD 3CEiWQo1rQAiPZxSXMjNcsgRfMsK8wm2HZOJs= Received: by 10.141.161.6 with SMTP id n6mr3849705rvo.57.1230053148942; Tue, 23 Dec 2008 09:25:48 -0800 (PST) Received: by 10.141.123.6 with HTTP; Tue, 23 Dec 2008 09:25:48 -0800 (PST) Message-ID: Date: Tue, 23 Dec 2008 12:25:48 -0500 From: "Kynn Jones" To: pgsql-performance@postgresql.org Subject: How to "unique-ify" HUGE table? MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_65946_26911458.1230053148930" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 200812/202 X-Sequence-Number: 32238 ------=_Part_65946_26911458.1230053148930 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi everyone! I have a very large 2-column table (about 500M records) from which I want to remove duplicate records. I have tried many approaches, but they all take forever. The table's definition consists of two short TEXT columns. It is a temporary table generated from a query: CREATE TEMP TABLE huge_table AS SELECT x, y FROM ... ; Initially I tried CREATE TEMP TABLE huge_table AS SELECT DISTINCT x, y FROM ... ; but after waiting for nearly an hour I aborted the query, and repeated it after getting rid of the DISTINCT clause. Everything takes forever with this monster! It's uncanny. Even printing it out to a file takes forever, let alone creating an index for it. Any words of wisdom on how to speed this up would be appreciated. TIA! Kynn ------=_Part_65946_26911458.1230053148930 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi everyone!

I have a very large 2-column table (about 500M records) from which I want to remove duplicate records.

I have tried many approaches, but they all take forever.

The table's definition consists of two short TEXT columns.  It is a temporary table generated from a query:

CREATE TEMP TABLE huge_table AS SELECT x, y FROM ... ;

Initially I tried

CREATE TEMP TABLE huge_table AS SELECT DISTINCT x, y FROM ... ;

but after waiting for nearly an hour I aborted the query, and repeated it after getting rid of the DISTINCT clause.

Everything takes forever with this monster!  It's uncanny.  Even printing it out to a file takes forever, let alone creating an index for it.

Any words of wisdom on how to speed this up would be appreciated.

TIA!

Kynn



------=_Part_65946_26911458.1230053148930--