Message-ID: <4BD58A00.7030905@gmx.net>
Date: Mon, 26 Apr 2010 14:41:36 +0200
From: Andreas <maps.on@gmx.net>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de;
	rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: silly sad <sad@bankir.ru>
CC: pgsql-sql@postgresql.org
Subject: Re: [SPAM]-D] How to find broken UTF-8 characters ?
References: <4BD4DA7B.3000603@gmx.net> <4BD5670A.803@bankir.ru>
In-Reply-To: <4BD5670A.803@bankir.ru>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

Am 26.04.2010 12:12, schrieb silly sad:
> On 04/26/10 04:12, Andreas wrote:
>
> looks like a complete offtopic
Not anymore. The bad signs are in the DB now.

I'd need some command that filters somehow for inconvertible 
(Unicode-->local charset) data.
How can I find those Unicode characters that allready sneaked in?

Actually there shouldn't be anything within the tables that NEED to be 
coded in Unicode.

something like
SELECT * FROM tab_1 WHERE  field_x <>  ConvertToLocal(field_x)
might be a good start.


>> How can I get rid of them?
> iconv -c
AFAIK iconv would translate on file system level but I would think that 
messed up a allready messed up Excel workmap even further.
I'd be glad to handle csv, too.

> BUT
> u should not have those characters at all
> if one is occured it most probably an error

Sure, but those files hit me over a chain of people who consider it ok 
to convert data over numerus file formats, cut, edit, save as X, send 
per mail .... then hit me and I am the one to clean up.


> AND
> u should get rid of this error itself -- not of its consequences.
Like quitting the job and grow flowers instead?
I'll consider this.   ;)