From: gnews84@hotmail.com
To: pgsql-sql@postgresql.org
Subject: Re: Removing duplicates
Date: 26 Feb 2002 20:52:49 -0800
Message-ID: <a03f6b0d.0202262052.794fa621@posting.google.com> (raw)
References: <5.1.0.14.2.20020226095955.00b17f10@imap.brwholesale.com>
Besides the obvious problems that you mention such as punctuation
spelling etc., there are also a few other things that you should keep
in mind:
- A lot of your customers may have moved in the past 8 years. In fact
I was reading somewhere that about 10% of US residents move on an
average of once a year. Especially apartment addresses in your
database are very unreliable.
- Phone numbers are even more unreliable because not only that they
change when a person moves but they also change when an area code
split occurs. Also in many cases people may give you a work phone.
-Women change Last Names when they marry, divorce etc.
Based on the budget and the number of records in you database you can
do different things to improve your data quality:
If you have the money to spend and the database is large, you can buy
some commercially available products such as First Logic’s ACE
http://www.firstlogic.com/home.asp
or Group One's "Code-1 Plus"
http://www.g1.com/g1intro.html
These products have a database of all addresses in the US standardized
by USPS standards. They can do "soft matches" on addresses for a given
zip code and then standardize the address. Or they can add zip and zip
+ 4 based on you street address and city. Or they can match state with
zip, county with zip, town with zip etc.
On top of that, you may want to NCOA your database. NCOA stands for
National change of address and it is a huge database that USPS keeps
with every address change that people made in the past (I am not sure
how far back it goes). You can find many 3rd party vendors that will
NCOA your database. Just go to goggle and do a search under "NCOA".
Be aware that the software can cost at least $20,000 and the NCOA may
cost 1 cent per address.
A cheaper option for a one time job would be to standardize and NCOA
you database through 3rd parties. I am not sure what it will cost but
if I find more info I will post it here.
The third option for improving your data quality is to use some
inexpensive third party databases to match your zip code with area
code and state. For example look at:
http://www.zipinfo.com/
In order to match customers you have to have an idea of how many
records you have on each zip code. If the number is small (less then
200-300) you can match on zip code, primary range and last name.
Primary range is the number before the address. (For example in
"1234 W. Main Street", primary range is 1234. In terms of SQL you need
to match on first 4 characters of Address1 field. If customers are
concentrated in one or a few zip codes then you probably need to match
on full address1.
Hope this helps.
Tony
PS. Did I tell you that I do consulting on data cleansing and
Merge/Purge :-)
gnews84@hotmail.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: gnews84@hotmail.com
Subject: Re: Removing duplicates
In-Reply-To: <a03f6b0d.0202262052.794fa621@posting.google.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox