X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id F02009DCAD0 for ; Sat, 11 Feb 2006 12:14:43 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 46642-06 for ; Sat, 11 Feb 2006 12:14:42 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id C3CAE9DCAA9 for ; Sat, 11 Feb 2006 12:14:41 -0400 (AST) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id k1BGEbqM008169; Sat, 11 Feb 2006 11:14:37 -0500 (EST) To: Mario Splivalo cc: pgsql-admin@postgresql.org Subject: Re: Pg 7.4 to 8.1 UTF problems In-reply-to: <1139151996.14901.8.camel@localhost.localdomain> References: <1139151996.14901.8.camel@localhost.localdomain> Comments: In-reply-to Mario Splivalo message dated "Sun, 05 Feb 2006 16:06:35 +0100" Date: Sat, 11 Feb 2006 11:14:37 -0500 Message-ID: <8168.1139674477@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.109 required=5 tests=[AWL=0.109] X-Spam-Score: 0.109 X-Spam-Level: X-Archive-Number: 200602/145 X-Sequence-Number: 20566 Mario Splivalo writes: > psql:the_database.psql:86: ERROR: invalid UTF-8 byte sequence detected > near byte 0x8d > CONTEXT: COPY _netsms, line 1367, column text: "Padamo u pozu 69.uz > \uffffmas mog macana u usta.a ja tvoju ljepu picu" 7.4's checking for valid UTF8 code sequences had some bugs, causing it to accept data that is not valid UTF8. 8.1 has tightened that up. > From here it is visible that the troubled charachter has has ASCII HEX > code 8d, which is regular ascii charachter. It is not ASCII, and it is not legal UTF8 either, at least not without another byte >= 0x80 after it. You need to decide whether this is bad data (and if so fix it), or whether you misdetermined what the encoding of your data is (and if so, change to the correct encoding declaration). regards, tom lane