To: "Rajesh Mallah" <mallah_rajesh@yahoo.com>,
	pgsql-sql@postgresql.org
Subject: Re: Significance of Database Encoding
References: <20050515183829.53021.qmail@web31011.mail.mud.yahoo.com>
Message-ID: <op.sqt1blpqth1vuj@localhost>
Date: Sun, 15 May 2005 21:48:47 +0200
From: PFC <lists@boutiquenumerique.com>
Organization: =?iso-8859-15?Q?La_Boutique_Num=E9rique?=
Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
In-Reply-To: <20050515183829.53021.qmail@web31011.mail.mud.yahoo.com>
User-Agent: Opera M2(BETA2)/8.0 (Linux, build 987)


> $ iconv -f US-ASCII -t UTF-8  < test.sql > out.sql
> iconv: illegal input sequence at position 114500
>
> Any ideas how the job can be accomplised reliably.
>
> Also my database may contain data in multiple encodings
> like WINDOWS-1251 and WINDOWS-1256 in various places
> as data has been inserted by different peoples using
> different sources and client software.

	You could use a simple program like that (in Python):

output = open( "unidump", "w" )
for line in open( "your dump" ):
	for encoding in "utf-8", "iso-8859-15", "whatever":
		try:
			output.write( unicode( line, encoding ).encode( "utf-8" ))
			break
		except UnicodeError:
			pass
	else:
		print "No suitable encoding for line..."

	I'd say this might work, if UTF-8 cannot absorb an apostrophe inside a  
multibit character. Can it ?

	Or you could do that to all your table using SELECTs but it's going to be  
painful...