pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Joel Fradkin <jfradkin@wazagua.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Salem Berhanu <salemb4@hotmail.com>
Cc: pgsql-admin@postgresql.org
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] postgres & server encodings
Date: Tue, 9 Aug 2005 13:18:53 -0400
Message-ID: <20050809171853.GB21039@alvh.no-ip.org> (raw)
In-Reply-To: <001601c59d03$4f107af0$797ba8c0@jfradkin>
References: <16705.1123603167@sss.pgh.pa.us>
	<001601c59d03$4f107af0$797ba8c0@jfradkin>

On Tue, Aug 09, 2005 at 12:56:37PM -0400, Joel Fradkin wrote:
> Not that I am an expert or anything, but my initial data base was SQLASCII
> and I did have to convert it to Unicode.
> My reasons were we store French characters in our database and the newer
> odbc driver was not displaying them correctly coming from SQLASCII, but was
> from UNICODE.
> I also think that it can affect functions like length and upper, but Tom
> knows a ton more then me about this stuff.
> 
> I did my initial conversion on 7.4 and the odbc driver at that time had no
> issues with SQLASCII displaying the French, but I think in 8.0.1 I started
> seeing an issue. The latest version of the driver 8.0.4 seems to be working
> well (only up a little over 24 hours thus far).

A conversion will work fine assuming the data is all encoded using the
same encoding.  So if it's all utf8 ("Unicode") already, you can import
it verbatim into a UTF8 database and it will work fine.  If it's all
Latin-1, you can import into a UTF-8 db using a client_encoding=latin1
during import, or verbatim to a Latin-1 database, and it will also work
fine.  (You of course are expected to be able to figure out what
encoding is the data really in.)

The problem only shows up when you have mixed data -- say, you have two
applications, one website in PHP which inserts data in Latin-1, and a
Windows app which inserts in UTF-8.  In this case your data will be a
mess to fix, and there's no way a single conversion will get it right.
You will have to manually separate the parts that are UTF8 from the
Latin1, and import them separately.  Not a position I'd like to be in.

-- 
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
"Coge la flor que hoy nace alegre, ufana. ¿Quién sabe si nacera otra mañana?"



view thread (9+ messages)  latest in thread

Message-ID: <20050809171853.GB21039@alvh.no-ip.org>
Permalink:  ../20050809171853.GB21039@alvh.no-ip.org/
Also on:    postgresql.org/message-id/20050809171853.GB21039@alvh.no-ip.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: alvherre@alvh.no-ip.org, jfradkin@wazagua.com, tgl@sss.pgh.pa.us, salemb4@hotmail.com, pgsql-general@postgresql.org
  Subject: Re: [GENERAL] postgres & server encodings
  In-Reply-To: <20050809171853.GB21039@alvh.no-ip.org>
  References: <16705.1123603167@sss.pgh.pa.us> <001601c59d03$4f107af0$797ba8c0@jfradkin> <20050809171853.GB21039@alvh.no-ip.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox