X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 39AEC529A5 for ; Tue, 9 Aug 2005 14:31:20 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 47084-10 for ; Tue, 9 Aug 2005 17:31:15 +0000 (GMT) Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by svr1.postgresql.org (Postfix) with ESMTP id 0D69D5288D for ; Tue, 9 Aug 2005 14:31:14 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j79HV39K017413; Tue, 9 Aug 2005 13:31:04 -0400 (EDT) To: Alvaro Herrera Cc: Joel Fradkin , "'Salem Berhanu'" , pgsql-admin@postgresql.org, pgsql-general@postgresql.org Subject: Re: [GENERAL] postgres & server encodings In-reply-to: <20050809171853.GB21039@alvh.no-ip.org> References: <16705.1123603167@sss.pgh.pa.us> <001601c59d03$4f107af0$797ba8c0@jfradkin> <20050809171853.GB21039@alvh.no-ip.org> Comments: In-reply-to Alvaro Herrera message dated "Tue, 09 Aug 2005 13:18:53 -0400" Date: Tue, 09 Aug 2005 13:31:03 -0400 Message-ID: <17412.1123608663@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.006 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200508/113 X-Sequence-Number: 18432 Alvaro Herrera writes: > The problem only shows up when you have mixed data -- say, you have two > applications, one website in PHP which inserts data in Latin-1, and a > Windows app which inserts in UTF-8. In this case your data will be a > mess to fix, and there's no way a single conversion will get it right. > You will have to manually separate the parts that are UTF8 from the > Latin1, and import them separately. Not a position I'd like to be in. The only helpful tip I can think of is that you can try to import data into a UTF8 database and see if it gets rejected as badly encoded; this will at least give you a weak tool to separate what's what. I'm afraid the reverse direction won't help much --- in single-byte encodings such as Latin1 there are no encoding errors, and so you can't do any simple filtering to check in that direction. In the end you're going to have to eyeball a lot of data for plausibility :-( regards, tom lane