Message-ID: <4DDB02E9.9040509@postnewspapers.com.au>
Date: Tue, 24 May 2011 08:59:21 +0800
From: Craig Ringer <craig@postnewspapers.com.au>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
	rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: Tarlika Elisabeth Schmitz <postgresql3@numerixtechnology.de>
CC: sql pgsql <pgsql-sql@postgresql.org>
Subject: Re: extracting location info from string
References: <20110522210526.0f8b66fa@dick.coachhouse>
	<20110522230041.28d1bc35@dick.coachhouse>
	<BANLkTikGzsrvXprLXeN3TxZwA=M8fzkn=g@mail.gmail.com>
	<20110523233949.07fb0005@dick.coachhouse>
In-Reply-To: <20110523233949.07fb0005@dick.coachhouse>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 24/05/2011 6:39 AM, Tarlika Elisabeth Schmitz wrote:

> Indeed. However, the situation is not quite as bleak as it appears:
> - I am only dealing with 50 countries (Luxemburg and Vatican are not
> amongst them)
> - Only for two countries will city/region be displayed instead of
> country.
> - Ultimately, where the only important bit of imformation is the
> country.
> - The only really important persons are those from the two countries.

Ah, see that's critical. You've just been able to restrict the problem 
domain to a much simpler task with a smaller and well-defined range of 
possibilities. Most of the complexity is in the nasty corner cases and 
weirdness, and you've (probably) just cut most of that away.

> Of 17000 historical records, 4400 don't match this simple pattern.
> Of the 4400, 1300 are "USA" or "Usa" instead of "United States", 900
> "North America" whatever that is! There are plenty of common +
> valid region abbreviations.
>
> I get about 1000 new records of this type per year.

I'd do this kind of analysis in a PL/Perl or PL/python function myself. 
It's easier to write "If <x> then <y> else <x>" logic in a readable 
form, and such chained tests are usually better for this sort of work. 
That also makes it easier to do a cleanup pass first, where you 
substitute common spelling errors and canonicalize country names.

> However, the import process has to be as automatic as possible in such
> a way that inconsistencies are flagged up for later manual
> intervention. I say later because, for instance, a person's data will
> have to be imported with or without location info because other new
> data will link to it.

That's another good reason to use a PL function for this cleanup work. 
It's easy to INSERT a record into a side table that flags it for later 
examination if necessary,  and to RAISE NOTICE or to issue a NOTIFY if 
you need to do closer-to-realtime checking.

-- 
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/