pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Paul Jungwirth <pj@illuminatedcomputing.com>
To: pgsql-general@lists.postgresql.org
Subject: Re: Need efficient way to remove (almost) duplicate rows from a table
Date: Tue, 17 Oct 2023 16:09:17 -0700
Message-ID: <c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com> (raw)
In-Reply-To: <CAEs=6D=Pox3tGHAVZdT_-v+H_o5qpJU0Eg=3wwQ1CBQj+oN3Zw@mail.gmail.com>
References: <CAEs=6D=Pox3tGHAVZdT_-v+H_o5qpJU0Eg=3wwQ1CBQj+oN3Zw@mail.gmail.com>

On 10/17/23 15:48, David Gauthier wrote:
> I have a table with millions of records and there are a lot of "almost" 
> duplicates that I want to get rid of in an efficient way.  Best to 
> illustrate with a simple example...

This sounds like a classic use-case for a "correlated subquery". 
Assuming you want to keep the lowest twig_id, you could do this:

```
paul=# select * from twigs;
  limb_id | branch_id | twig_id | surfarea
---------+-----------+---------+----------
        1 |         1 |       1 |      100
        1 |         1 |       2 |    103.7
        1 |         1 |       3 |    103.7
        1 |         1 |       4 |    110.4
        1 |         2 |       1 |      120
        1 |         2 |       2 |    123.6
        1 |         2 |       3 |    128.1
        1 |         2 |       4 |    128.1
        1 |         2 |       5 |    128.1
        2 |         1 |       1 |      100
        2 |         1 |       3 |    104.4
        2 |         1 |       4 |    131.9
(12 rows)

Time: 0.218 ms
paul=# delete from twigs as t1
where exists (select 1 from twigs as t2 where (t1.limb_id, t1.branch_id, 
t1.surfarea) = (t2.limb_id, t2.branch_id, t2.surfarea) and t1.twig_id > 
t2.twig_id);
DELETE 3
Time: 8.555 ms
paul=# select * from twigs;
  limb_id | branch_id | twig_id | surfarea
---------+-----------+---------+----------
        1 |         1 |       1 |      100
        1 |         1 |       2 |    103.7
        1 |         1 |       4 |    110.4
        1 |         2 |       1 |      120
        1 |         2 |       2 |    123.6
        1 |         2 |       3 |    128.1
        2 |         1 |       1 |      100
        2 |         1 |       3 |    104.4
        2 |         1 |       4 |    131.9
(9 rows)

Time: 0.231 ms
```

That says "delete from t1 where there is a matching t2 with a lower 
twig_id."

If you want to keep the highest-numbered twig_id, the sql is easy to adjust.

Regards,

-- 
Paul              ~{:-)
pj@illuminatedcomputing.com





view thread (4+ messages)  latest in thread

Message-ID: <c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com>
Permalink:  ../c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com/
Also on:    postgresql.org/message-id/c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: pj@illuminatedcomputing.com
  Subject: Re: Need efficient way to remove (almost) duplicate rows from a table
  In-Reply-To: <c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com>
  References: <CAEs=6D=Pox3tGHAVZdT_-v+H_o5qpJU0Eg=3wwQ1CBQj+oN3Zw@mail.gmail.com> <c6e4f48d-3af4-4d65-4031-b1e447e09214@illuminatedcomputing.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox