Received: from mail.gl.aser.de (s217-115-138-162.colo.hosteurope.de [217.115.138.162]) by postgresql.org (Postfix) with ESMTP id 5159B476F91 for ; Tue, 26 Feb 2002 12:25:23 -0500 (EST) Received: from there (dialin-145-254-175-222.arcor-ip.net [145.254.175.222]) by mail.gl.aser.de (Postfix) with SMTP id 7B8FF636A3; Tue, 26 Feb 2002 18:19:44 +0100 (CET) Content-Type: text/plain; charset="iso-8859-1" From: Christof Glaser Organization: gl.aser . software engineering . internet service To: Matthew Hagerty , pgsql-sql@postgresql.org Subject: Re: Removing duplicates Date: Tue, 26 Feb 2002 18:26:34 +0100 X-Mailer: KMail [version 1.3.2] References: <5.1.0.14.2.20020226095955.00b17f10@imap.brwholesale.com> In-Reply-To: <5.1.0.14.2.20020226095955.00b17f10@imap.brwholesale.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-Id: <20020226171944.7B8FF636A3@mail.gl.aser.de> X-Archive-Number: 200202/405 X-Sequence-Number: 6827 Matthew, On Tuesday, 26. February 2002 16:10, Matthew Hagerty wrote: > Greetings, > > I have a customer database (name, address1, address2, city, state, zip) > and I need a query (or two) that will give me a mailing list with the > least amount of duplicates possible. I know that precise matching is > not possible, i.e. "P.O. Box 123" will never match "PO Box 123" without > some data massaging, but if I can isolate even 50% of any duplicates, > that would help greatly. > > Also, any suggestions on which parameters to check the duplicates for? > My first thoughts were to make sure there were no two addresses the > same in the same zip code. Any insight (or examples) would be greatly > appreciated. I wrote a (Windows) C++ program some time ago to eliminate duplicate address records in a plain text file. Basically it works like this: 1. Choose the fields to check for duplicates. Let's call these primary fields. I used Name, State, ZIP, City, Address1, Address2 (in that order). 2. Choose the fields to decide, which entry should be kept, if duplicates are found. These are secondary fields. (phone, contact, email, ...) 3. Normalize the primary fields: remove any whitespace, punctuation, special chars, make lowercase, expand "St." to "street" etc. That makes "P.O. Box 123" equal to "PO Box 123", as both are normalized to "pobox123". 4. Sort on the normalized primary fields. Now duplicates are adjacent. 5. Iterate through the sorted data: - Compare (normalized) primary fields with the next record - If equal, we found a duplicate and start another iteration: . Check the secondary fields and decide, which record to keep. (In my case, those with a phone # would be kept.) . Compare primary fields again with the next record, checking for more duplicates Translating the above into SQL is left as an exercise... :-) Hope that helps to get you started. Christof. -- gl.aser . software engineering . internet service http://gl.aser.de/ . Planckstraße 7 . D-39104 Magdeburg