Message-ID: <b12ef47c4298ca05a9e4b159ce7c71910e0206ea.camel@cybertec.at>
Subject: Re: database encoding and collation.
From: Laurenz Albe <laurenz.albe@cybertec.at>
To: "James Pang (chaolpan)" <chaolpan@cisco.com>,
	"pgsql-admin@lists.postgresql.org"
	 <pgsql-admin@lists.postgresql.org>
Date: Fri, 14 Jan 2022 08:17:14 +0100
In-Reply-To: 
 <PH0PR11MB5191393FFF0FDF0E510CF18CD6549@PH0PR11MB5191.namprd11.prod.outlook.com>
References: 
	<PH0PR11MB5191DBEC457323D03B5C4592D6529@PH0PR11MB5191.namprd11.prod.outlook.com>
	 <68331b01ee5ed0d02158747bf0e2d2e4476cbbb8.camel@cybertec.at>
	 <PH0PR11MB5191393FFF0FDF0E510CF18CD6549@PH0PR11MB5191.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.40.4 (3.40.4-2.fc34) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/b12ef47c4298ca05a9e4b159ce7c71910e0206ea.camel%40cybertec.at>
Precedence: bulk

On Fri, 2022-01-14 at 02:11 +0000, James Pang (chaolpan) wrote:
> This question is about how to handle multiple languages in database , English, Japanese ,Chinese...
> The Oracle database used WE8ISO8859P1 from long time ago , when put "Chinese" letters in database,
> client application just use  WE8ISO8859P1 (client encoding) to insert data into database ,
> so no conversion there. When client app query the data , it still get the data with WE8ISO8859P1 ,
> and then convert that data to UTF8 from client app side.   Now, we plan migrate to Postgresql 13,
> want Postgresql database use WE8ISO8859P1 same as Oracle too, and same handling as Oracle. 
> Any potential risk ?

I see the problem: the sloppy checking of Oracle allowed you to insert corrupted strings into
the Oracle database by setting client encoding equal to server encoding, and you might not
detect the problem until you try to read the data with a different client encoding.

You may be able to carry on like this in PostgreSQL, since all bytes are valid LATIN1
code points, but (like in Oracle) you will run into problems as soon as you access the
database with a different client encoding, for example with a JDBC driver.

It would be better to fix the problem.  If all the data in the database are actually
encoded in a certain encoding (different from ISO-8859-1), you could simply import them
into a database with that encoding, and you would have the benefit of having correct data.
If different strings are encoded in different encodings, then you would have to repair the
data before doing that.

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com