From: Volkan YAZICI <yazicivo@ttmail.com>
To: Andreas <maps.on@gmx.net>
Cc: pgsql-sql@postgresql.org
Subject: Re: How to find double entries
Date: Wed, 16 Apr 2008 09:15:42 +0300
Message-ID: <87zlru72b5.fsf@alamut.mobiliz.com.tr> (raw)
In-Reply-To: <48056F5F.30507@gmx.net>
References: <48056F5F.30507@gmx.net>
On Wed, 16 Apr 2008, Andreas <maps.on@gmx.net> writes:
> how can I find double entries in varchar columns where the content is
> not 100% identical because of a spelling error or the person
> considered it "looked nicer" that way?
>
> I'd like to identify and then merge records of e.g. 'google',
> gogle', 'guugle'
>
> Then I want to match abbrevations like 'A-Company Ltd.', 'a company
> ltd.', 'A-Company Limited'
>
> Is there a way to do this?
> It would be OK just to list candidats up to be manually checked
> afterwards.
You can try something similar to below example. (levenshtein(text, text)
function is supplied by fuzzystrmatch module.)
SELECT T1.col, T2.col
FROM tbl AS T1,
INNER JOIN tbl AS T2
ON T1.col <> T2.col AND
levenshtein(T1.col, T2.col) < (length(T1.col) * 0.5)
Regards.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: yazicivo@ttmail.com, maps.on@gmx.net
Subject: Re: How to find double entries
In-Reply-To: <87zlru72b5.fsf@alamut.mobiliz.com.tr>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox