pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Character encoding problems and dump import
4+ messages / 3 participants
[nested] [flat]

* Character encoding problems and dump import
@ 2006-03-20 20:21 ogjunk-pgjedan@yahoo.com
  2006-03-20 22:31 ` Re: Character encoding problems and dump import Ivo Rossacher <rossacher@bluewin.ch>
  2006-03-21 05:31 ` Re: Character encoding problems and dump import John DeSoi <desoi@pgedit.com>
  0 siblings, 2 replies; 4+ messages in thread

From: ogjunk-pgjedan@yahoo.com @ 2006-03-20 20:21 UTC (permalink / raw)
  To: pgsql-admin

Hello,

I have a dump (non-binary, if it matters) of a DB that has some characters in it that my DB doesn't want to take.
I'm using PG 8.0.3 and it was created with Unicode support:

=> \encoding
UNICODE

Characters that cause problems during the import are things like:
é and other characters from the Extended ASCII table (c.f. bottom of http://www.lookuptables.com/ )
Also:
'ÇѱÛÀÌ Á¦´ë·Î µÇ´ÂÁö ½ÇÇè... ¿©±â¿¡´Â ¾Æ¹« ¸µÅ©µµ ¾ø½À´Ï´Ù.±×³É ÀÌ ³»¿ë¹Û¿¡ ¾ø½À´Ï´Ù.'


The errors I get on import are of this type:
  ERROR:  invalid byte sequence for encoding "UNICODE": 0xdb20

The data may not be the cleanest, and I have limited control over that.
But I am wondering if there is any way I can import this data, even if that means converting some of the characters into something else.

Thanks,
Otis





^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Character encoding problems and dump import
  2006-03-20 20:21 Character encoding problems and dump import ogjunk-pgjedan@yahoo.com
@ 2006-03-20 22:31 ` Ivo Rossacher <rossacher@bluewin.ch>
  1 sibling, 0 replies; 4+ messages in thread

From: Ivo Rossacher @ 2006-03-20 22:31 UTC (permalink / raw)
  To: pgsql-admin; +Cc: ogjunk-pgjedan@yahoo.com

Am Montag, 20. März 2006 21.21 schrieb ogjunk-pgjedan@yahoo.com:
> Hello,
>
> I have a dump (non-binary, if it matters) of a DB that has some characters
> in it that my DB doesn't want to take. I'm using PG 8.0.3 and it was
> created with Unicode support:
>
> => \encoding
> UNICODE

This is the client encoding (see \?).
To get server encoding you can do
show server_encoding;  (see command show in the command reference)

Then have a look at the dump to check what the encoding was when the dump was 
taken. (there is a line like set client_encoding = .... somewhere at the 
beginning of the dump)

There where some changes within the unicode handling some time ago. If the 
dump was taken by an other server version there migth be differences. (search 
the archives, there are serveral threads about the issue.

Best regards
Ivo

>
> Characters that cause problems during the import are things like:
> é and other characters from the Extended ASCII table (c.f. bottom of
> http://www.lookuptables.com/ ) Also:
> 'ÇѱÛÀÌ Á¦´ë·Î µÇ´ÂÁö ½ÇÇè... ¿©±â¿¡´Â ¾Æ¹« ¸µÅ©µµ ¾ø½À´Ï´Ù.±×³É ÀÌ
> ³»¿ë¹Û¿¡ ¾ø½À´Ï´Ù.'
>
>
> The errors I get on import are of this type:
>   ERROR:  invalid byte sequence for encoding "UNICODE": 0xdb20
>
> The data may not be the cleanest, and I have limited control over that.
> But I am wondering if there is any way I can import this data, even if that
> means converting some of the characters into something else.
>
> Thanks,
> Otis
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Character encoding problems and dump import
  2006-03-20 20:21 Character encoding problems and dump import ogjunk-pgjedan@yahoo.com
@ 2006-03-21 05:31 ` John DeSoi <desoi@pgedit.com>
  2006-03-21 14:46   ` Re: Character encoding problems and dump import ogjunk-pgjedan@yahoo.com
  1 sibling, 1 reply; 4+ messages in thread

From: John DeSoi @ 2006-03-21 05:31 UTC (permalink / raw)
  To: ogjunk-pgjedan@yahoo.com; ogjunk-pgjedan@yahoo.com; +Cc: pgsql-admin


On Mar 20, 2006, at 3:21 PM, <ogjunk-pgjedan@yahoo.com> <ogjunk- 
pgjedan@yahoo.com> wrote:

> The data may not be the cleanest, and I have limited control over  
> that.
> But I am wondering if there is any way I can import this data, even  
> if that means converting some of the characters into something else.

inconv might be able to help you fix encoding problems

http://www.gnu.org/software/libiconv/documentation/libiconv/iconv.1.html



John DeSoi, Ph.D.
http://pgedit.com/
Power Tools for PostgreSQL




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Character encoding problems and dump import
  2006-03-20 20:21 Character encoding problems and dump import ogjunk-pgjedan@yahoo.com
  2006-03-21 05:31 ` Re: Character encoding problems and dump import John DeSoi <desoi@pgedit.com>
@ 2006-03-21 14:46   ` ogjunk-pgjedan@yahoo.com
  0 siblings, 0 replies; 4+ messages in thread

From: ogjunk-pgjedan@yahoo.com @ 2006-03-21 14:46 UTC (permalink / raw)
  To: pgsql-admin

Thanks John and Ivo for help.
It turned out that I had to manually SET CLIENT_ENCODING TO 'LATIN1' before processing the dump (which didn't have this specified).  This fixed the problem.

I thought a DB set to UNICODE char encoding (server_encoding) would process the Extended ASCII characters, but it didn't... not sure why.

Otis

----- Original Message ----
From: John DeSoi <desoi@pgedit.com>
To: ogjunk-pgjedan@yahoo.com
Cc: pgsql-admin@postgresql.org
Sent: Tuesday, March 21, 2006 12:31:16 AM
Subject: Re: [ADMIN] Character encoding problems and dump import


On Mar 20, 2006, at 3:21 PM, <ogjunk-pgjedan@yahoo.com> <ogjunk- 
pgjedan@yahoo.com> wrote:

> The data may not be the cleanest, and I have limited control over  
> that.
> But I am wondering if there is any way I can import this data, even  
> if that means converting some of the characters into something else.

inconv might be able to help you fix encoding problems

http://www.gnu.org/software/libiconv/documentation/libiconv/iconv.1.html



John DeSoi, Ph.D.
http://pgedit.com/
Power Tools for PostgreSQL


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org






^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2006-03-21 14:46 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-03-20 20:21 Character encoding problems and dump import ogjunk-pgjedan@yahoo.com
2006-03-20 22:31 ` Ivo Rossacher <rossacher@bluewin.ch>
2006-03-21 05:31 ` John DeSoi <desoi@pgedit.com>
2006-03-21 14:46   ` ogjunk-pgjedan@yahoo.com

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox