X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 7E3429FB1B7 for ; Tue, 25 Jul 2006 23:22:49 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 30142-07 for ; Tue, 25 Jul 2006 23:22:39 -0300 (ADT) X-Greylist: delayed 00:06:40.260745 by SQLgrey- Received: from web56603.mail.re3.yahoo.com (web56603.mail.re3.yahoo.com [66.196.97.47]) by postgresql.org (Postfix) with SMTP id EF7279FA48B for ; Tue, 25 Jul 2006 23:22:39 -0300 (ADT) Received: (qmail 89939 invoked by uid 60001); 26 Jul 2006 02:15:57 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=FA5YqTj5idwc7XlG6oB3TCO4FZycRGG+eOOg0ebuWHC39We7pz9kKmk+HolKi+q2mZPwCmzx6/yCjB0LOM8rj28rItfVSuZ0qmHzkBahct9t9nsc6JAq4mdDkTpVksDJXckbKyBNx3cEB8kiyv+LByG1uv1QD/iuJAVN148qcXA= ; Message-ID: <20060726021557.89937.qmail@web56603.mail.re3.yahoo.com> Received: from [203.143.27.130] by web56603.mail.re3.yahoo.com via HTTP; Tue, 25 Jul 2006 19:15:57 PDT Date: Tue, 25 Jul 2006 19:15:57 -0700 (PDT) From: Thusitha Kodikara Subject: Re: invalid UTF-8 byte sequences and iconv To: Ivo Rossacher , pgsql-admin@postgresql.org In-Reply-To: <200607252158.47484.rossacher@bluewin.ch> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-1503691828-1153880157=:83677" Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=4.707 tagged_above=0 required=5 tests=DNS_FROM_RFC_ABUSE, DNS_FROM_RFC_POST, DNS_FROM_RFC_WHOIS, HTML_10_20, HTML_MESSAGE X-Spam-Level: **** X-Archive-Number: 200607/217 X-Sequence-Number: 22463 --0-1503691828-1153880157=:83677 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hello, We had a similar problem a few weeks back converting from PostgreSQL 7.3.10 to 7.4.13. After trying various methods, including iconv, we found that the one that worked in our case was to manually fix the data - which was only about 15 records fortunately. BTW thanks Ivo for suggesing that solution at that instance also :) Regards, -Thusitha Ivo Rossacher wrote: In earlier version of postgres the database did allow to store invalid byte sequences. The newer versions do check correctly for the byte sequences and do not allow invalid sequences. So if your dump is really in UTF8 already you will have to search for the invalid sequences in the dump and replace them with the correct one. (if you have a lot of them and a big dump recode might by of help for you). If the dump is not UTF8 you have to pass the correct encoding to iconv in the procedure you described. Best regards Ivo Am Dienstag, 25. Juli 2006 21.04 schrieb Karen Springer: > Hi, > > We have set up a new server and are needing to move our database from > 7.3 to 8.1.4. On restore I'm getting the 'invalid UTF-8 byte sequence' > error message. If I use the command iconv -c -f UTF-8 -t UTF-8 -o > cleanfile.sql dumpfile.sql, then the characters are deleted and the > restore goes smoothly. The problem is that we want those characters. > They are for example the degree symbol and the micro symbol. Is there > anyway to bring these characters over? Thanks in advance. > > Karen > > ---------------------------(end of broadcast)--------------------------- > TIP 5: don't forget to increase your free space map settings ---------------------------(end of broadcast)--------------------------- TIP 2: Don't 'kill -9' the postmaster --0-1503691828-1153880157=:83677 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hello,

We had a similar problem a few weeks back converting from PostgreSQL 7.3.10 to 7.4.13. After trying various methods, including iconv, we found that the one that worked in our case was to manually fix the data - which was only about 15 records fortunately.

BTW thanks Ivo for suggesing that solution at that instance also :)

Regards,
-Thusitha


Ivo Rossacher <rossacher@bluewin.ch> wrote:
In earlier version of postgres the database did allow to store invalid byte
sequences. The newer versions do check correctly for the byte sequences and
do not allow invalid sequences. So if your dump is really in UTF8 already you
will have to search for the invalid sequences in the dump and replace them
with the correct one. (if you have a lot of them and a big dump recode might
by of help for you). If the dump is not UTF8 you have to pass the correct
encoding to iconv in the procedure you described.

Best regards
Ivo
Am Dienstag, 25. Juli 2006 21.04 schrieb Karen Springer:
> Hi,
>
> We have set up a new server and are needing to move our database from
> 7.3 to 8.1.4. On restore I'm getting the 'invalid UTF-8 byte sequence'
> error message. If I use the command iconv -c -f UTF-8 -t UTF-8 -o
> cleanfile.sql dumpfile.sql, then the characters are deleted and the
> restore goes smoothly. The problem is that we want those characters.
> They are for example the degree symbol and the micro symbol. Is there
> anyway to bring these characters over? Thanks in advance.
>
> Karen
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

--0-1503691828-1153880157=:83677--