pg.ddx.io  pgsql-sql@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Christof Glaser <gcg@gl.aser.de>
To: Matthew Hagerty <matthew@brwholesale.com>
To: pgsql-sql@postgresql.org
Subject: Re: Removing duplicates
Date: Tue, 26 Feb 2002 18:26:34 +0100
Message-ID: <20020226171944.7B8FF636A3@mail.gl.aser.de> (raw)
In-Reply-To: <5.1.0.14.2.20020226095955.00b17f10@imap.brwholesale.com>
References: <5.1.0.14.2.20020226095955.00b17f10@imap.brwholesale.com>

Matthew,

On Tuesday, 26. February 2002 16:10, Matthew Hagerty wrote:
> Greetings,
>
> I have a customer database (name, address1, address2, city, state, zip)
> and I need a query (or two) that will give me a mailing list with the
> least amount of duplicates possible.  I know that precise matching is
> not possible, i.e. "P.O. Box 123" will never match "PO Box 123" without
> some data massaging, but if I can isolate even 50% of any duplicates,
> that would help greatly.
>
> Also, any suggestions on which parameters to check the duplicates for? 
> My first thoughts were to make sure there were no two addresses the
> same in the same zip code.  Any insight (or examples) would be greatly
> appreciated.

I wrote a (Windows) C++ program some time ago to eliminate duplicate 
address records in a plain text file. Basically it works like this:

1. Choose the fields to check for duplicates. Let's call these primary
   fields. I used Name, State, ZIP, City, Address1, Address2 
   (in that order).
2. Choose the fields to decide, which entry should be kept, if duplicates
   are found. These are secondary fields. (phone, contact, email, ...)
3. Normalize the primary fields: remove any whitespace, punctuation,
   special chars, make lowercase, expand "St." to "street" etc. 
   That makes "P.O. Box 123" equal to "PO Box 123", as both are
   normalized to "pobox123".
4. Sort on the normalized primary fields. Now duplicates are adjacent.
5. Iterate through the sorted data:
   - Compare (normalized) primary fields with the next record
   - If equal, we found a duplicate and start another iteration:
     . Check the secondary fields and decide, which record to keep.
       (In my case, those with a phone # would be kept.)
     . Compare primary fields again with the next record, checking
       for more duplicates

Translating the above into SQL is left as an exercise... :-) 

Hope that helps to get you started.

Christof.
-- 
    	  gl.aser . software engineering . internet service
       http://gl.aser.de/ . Planckstraße 7 . D-39104 Magdeburg



view thread (8+ messages)  latest in thread

Message-ID: <20020226171944.7B8FF636A3@mail.gl.aser.de>
Permalink:  ../20020226171944.7B8FF636A3@mail.gl.aser.de/
Also on:    postgresql.org/message-id/20020226171944.7B8FF636A3@mail.gl.aser.de

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-sql@postgresql.org
  Cc: gcg@gl.aser.de, matthew@brwholesale.com
  Subject: Re: Removing duplicates
  In-Reply-To: <20020226171944.7B8FF636A3@mail.gl.aser.de>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox