pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
invalid UTF-8 byte sequences and iconv
4+ messages / 4 participants
[nested] [flat]

* invalid UTF-8 byte sequences and iconv
@ 2006-07-25 19:04 Karen Springer <karen.springer@wulfsberg.com>
  2006-07-25 19:29 ` Re: invalid UTF-8 byte sequences and iconv Alvaro Herrera <alvherre@commandprompt.com>
  2006-07-25 19:58 ` Re: invalid UTF-8 byte sequences and iconv Ivo Rossacher <rossacher@bluewin.ch>
  0 siblings, 2 replies; 4+ messages in thread

From: Karen Springer @ 2006-07-25 19:04 UTC (permalink / raw)
  To: pgsql-admin

Hi,

We have set up a new server and are needing to move our database from 
7.3 to 8.1.4.  On restore I'm getting the 'invalid UTF-8 byte sequence' 
error message.  If I use the command iconv -c -f UTF-8 -t UTF-8 -o 
cleanfile.sql dumpfile.sql, then the characters are deleted and the 
restore goes smoothly.  The problem is that we want those characters. 
They are for example the degree symbol and the micro symbol.  Is there 
anyway to bring these characters over?  Thanks in advance.

Karen



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: invalid UTF-8 byte sequences and iconv
  2006-07-25 19:04 invalid UTF-8 byte sequences and iconv Karen Springer <karen.springer@wulfsberg.com>
@ 2006-07-25 19:29 ` Alvaro Herrera <alvherre@commandprompt.com>
  1 sibling, 0 replies; 4+ messages in thread

From: Alvaro Herrera @ 2006-07-25 19:29 UTC (permalink / raw)
  To: Karen Springer <karen.springer@wulfsberg.com>; +Cc: pgsql-admin

Karen Springer wrote:
> Hi,
> 
> We have set up a new server and are needing to move our database from 
> 7.3 to 8.1.4.  On restore I'm getting the 'invalid UTF-8 byte sequence' 
> error message.  If I use the command iconv -c -f UTF-8 -t UTF-8 -o 
> cleanfile.sql dumpfile.sql, then the characters are deleted and the 
> restore goes smoothly.  The problem is that we want those characters. 
> They are for example the degree symbol and the micro symbol.  Is there 
> anyway to bring these characters over?  Thanks in advance.

Huh, maybe using the real source encoding instead?  Try, for example,
using Latin-1.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: invalid UTF-8 byte sequences and iconv
  2006-07-25 19:04 invalid UTF-8 byte sequences and iconv Karen Springer <karen.springer@wulfsberg.com>
@ 2006-07-25 19:58 ` Ivo Rossacher <rossacher@bluewin.ch>
  2006-07-26 02:15   ` Re: invalid UTF-8 byte sequences and iconv Thusitha Kodikara <kthusi@yahoo.com>
  1 sibling, 1 reply; 4+ messages in thread

From: Ivo Rossacher @ 2006-07-25 19:58 UTC (permalink / raw)
  To: pgsql-admin

In earlier version of postgres the database did allow to store invalid byte 
sequences. The newer versions do check correctly for the byte sequences and 
do not allow invalid sequences. So if your dump is really in UTF8 already you 
will have to search for the invalid sequences in the dump and replace them 
with the correct one. (if you have a lot of them and a big dump recode might 
by of help for you). If the dump is not UTF8 you have to pass the correct 
encoding to iconv in the procedure you described.

Best regards
Ivo
Am Dienstag, 25. Juli 2006 21.04 schrieb Karen Springer:
> Hi,
>
> We have set up a new server and are needing to move our database from
> 7.3 to 8.1.4.  On restore I'm getting the 'invalid UTF-8 byte sequence'
> error message.  If I use the command iconv -c -f UTF-8 -t UTF-8 -o
> cleanfile.sql dumpfile.sql, then the characters are deleted and the
> restore goes smoothly.  The problem is that we want those characters.
> They are for example the degree symbol and the micro symbol.  Is there
> anyway to bring these characters over?  Thanks in advance.
>
> Karen
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: invalid UTF-8 byte sequences and iconv
  2006-07-25 19:04 invalid UTF-8 byte sequences and iconv Karen Springer <karen.springer@wulfsberg.com>
  2006-07-25 19:58 ` Re: invalid UTF-8 byte sequences and iconv Ivo Rossacher <rossacher@bluewin.ch>
@ 2006-07-26 02:15   ` Thusitha Kodikara <kthusi@yahoo.com>
  0 siblings, 0 replies; 4+ messages in thread

From: Thusitha Kodikara @ 2006-07-26 02:15 UTC (permalink / raw)
  To: Ivo Rossacher <rossacher@bluewin.ch>; pgsql-admin

Hello,

We had a similar problem a few weeks back converting from PostgreSQL 7.3.10 to 7.4.13. After trying various methods, including iconv, we found that the one that worked in our case was to manually fix the data - which was only about 15 records fortunately.

BTW thanks Ivo for suggesing that solution at that instance also :)

Regards,
-Thusitha


Ivo Rossacher <rossacher@bluewin.ch> wrote: In earlier version of postgres the database did allow to store invalid byte 
sequences. The newer versions do check correctly for the byte sequences and 
do not allow invalid sequences. So if your dump is really in UTF8 already you 
will have to search for the invalid sequences in the dump and replace them 
with the correct one. (if you have a lot of them and a big dump recode might 
by of help for you). If the dump is not UTF8 you have to pass the correct 
encoding to iconv in the procedure you described.

Best regards
Ivo
Am Dienstag, 25. Juli 2006 21.04 schrieb Karen Springer:
> Hi,
>
> We have set up a new server and are needing to move our database from
> 7.3 to 8.1.4.  On restore I'm getting the 'invalid UTF-8 byte sequence'
> error message.  If I use the command iconv -c -f UTF-8 -t UTF-8 -o
> cleanfile.sql dumpfile.sql, then the characters are deleted and the
> restore goes smoothly.  The problem is that we want those characters.
> They are for example the degree symbol and the micro symbol.  Is there
> anyway to bring these characters over?  Thanks in advance.
>
> Karen
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2006-07-26 02:15 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-07-25 19:04 invalid UTF-8 byte sequences and iconv Karen Springer <karen.springer@wulfsberg.com>
2006-07-25 19:29 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-07-25 19:58 ` Ivo Rossacher <rossacher@bluewin.ch>
2006-07-26 02:15   ` Thusitha Kodikara <kthusi@yahoo.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox