pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Going from a DB using SQL_ASCII to UNICODE
7+ messages / 5 participants
[nested] [flat]

* Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-19 19:44 Dion Almaer <dion@almaer.com>
  2004-04-19 21:56 ` Re: Going from a DB using SQL_ASCII to UNICODE CoL <col@mportal.hu>
  2004-04-19 22:02 ` Re: Going from a DB using SQL_ASCII to UNICODE CoL <col@mportal.hu>
  2004-04-20 01:20 ` Re: Going from a DB using SQL_ASCII to UNICODE Tom Lane <tgl@sss.pgh.pa.us>
  2004-04-20 05:05 ` Re: Going from a DB using SQL_ASCII to UNICODE Frank Finner <postgresql@finner.de>
  0 siblings, 4 replies; 7+ messages in thread

From: Dion Almaer @ 2004-04-19 19:44 UTC (permalink / raw)
  To: pgsql-admin; +Cc: 'Lloyd Benson' <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>


Hi -

We are running PostgreSQL 7.1.4, and are finally upgrading... to 7.4.2.

While we do this upgrade, we also want to make create our DB as UNICODE (we
are having problems with SQL_ASCII where we get a bunch of ? marks).

When we try to import the data from a SQL_ASCII dumped db, into the new
UNICODE db, everything freaks out.
We get errors restoring from dumps that used COPY and with INSERT.

We also tried to use the 7.4 pg_dump to make sure that it wasn't a problem
with the old pg_dump.

The errors are: 

- ERROR:  missing data for column "noiselevel"
- CONTEXT:  COPY messages, line 1: "4393  -1      1441    14147   0
2000-10-12 18:30:58-05  EJB Design Questions    \N      Hi,"
- FATAL:  invalid frontend message type 60
- ERROR:  invalid byte sequence for encoding "UNICODE": 0xe9616c

Anyway, if anyone has any recommendations on how to migrate from SQL_ASCII
to UNICODE, with the knowledge that we are going from 7.1 to 7.4 at the same
time.... You will be life-savers!

Cheers,

Dion




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
@ 2004-04-19 21:56 ` CoL <col@mportal.hu>
  3 siblings, 0 replies; 7+ messages in thread

From: CoL @ 2004-04-19 21:56 UTC (permalink / raw)
  To: pgsql-admin

hi,

Dion Almaer wrote, On 4/19/2004 21:44:
> - ERROR:  missing data for column "noiselevel"
> - CONTEXT:  COPY messages, line 1: "4393  -1      1441    14147   0
> 2000-10-12 18:30:58-05  EJB Design Questions    \N      Hi,"

for this one: better to dump with: -d

C.



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
@ 2004-04-19 22:02 ` CoL <col@mportal.hu>
  3 siblings, 0 replies; 7+ messages in thread

From: CoL @ 2004-04-19 22:02 UTC (permalink / raw)
  To: pgsql-admin

hi,

Dion Almaer wrote, On 4/19/2004 21:44:
> - ERROR:  invalid byte sequence for encoding "UNICODE": 0xe9616c

and for this: convert your dump with iconv to unicode, then load it.

C.



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
@ 2004-04-20 01:20 ` Tom Lane <tgl@sss.pgh.pa.us>
  2004-04-20 01:24   ` Re: Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
  3 siblings, 1 reply; 7+ messages in thread

From: Tom Lane @ 2004-04-20 01:20 UTC (permalink / raw)
  To: Dion Almaer <dion@almaer.com>; +Cc: pgsql-admin@postgresql.org, "'Lloyd Benson'" <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>

"Dion Almaer" <dion@almaer.com> writes:
> When we try to import the data from a SQL_ASCII dumped db, into the new
> UNICODE db, everything freaks out.

I'm no expert on this stuff, but I think what you need to do is add

	set client_encoding = sql_ascii;

to the top of the dump file.  (As of fairly recently, pg_dump will
automatically add such a SET, but I'm pretty sure 7.4.2 won't.)

If that doesn't help, then what you have is actually not valid UTF-8
data, and what you'll have to do is figure out what encoding it's in and
specify that instead.  If it's in a mishmash of different encodings,
you're in for some pain :-(

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
  2004-04-20 01:20 ` Re: Going from a DB using SQL_ASCII to UNICODE Tom Lane <tgl@sss.pgh.pa.us>
@ 2004-04-20 01:24   ` Dion Almaer <dion@almaer.com>
  0 siblings, 0 replies; 7+ messages in thread

From: Dion Almaer @ 2004-04-20 01:24 UTC (permalink / raw)
  To: 'Tom Lane' <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin; Lloyd Benson <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>

Tom,

I will try to add "set client_encoding = sql_ascii;" and pray :)

Cheers,

Dion

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
> Sent: Monday, April 19, 2004 9:21 PM
> To: Dion Almaer
> Cc: pgsql-admin@postgresql.org; 'Lloyd Benson'; Eric Preston
> Subject: Re: [ADMIN] Going from a DB using SQL_ASCII to UNICODE 
> 
> "Dion Almaer" <dion@almaer.com> writes:
> > When we try to import the data from a SQL_ASCII dumped db, into the 
> > new UNICODE db, everything freaks out.
> 
> I'm no expert on this stuff, but I think what you need to do is add
> 
> 	set client_encoding = sql_ascii;
> 
> to the top of the dump file.  (As of fairly recently, pg_dump 
> will automatically add such a SET, but I'm pretty sure 7.4.2 won't.)
> 
> If that doesn't help, then what you have is actually not 
> valid UTF-8 data, and what you'll have to do is figure out 
> what encoding it's in and specify that instead.  If it's in a 
> mishmash of different encodings, you're in for some pain :-(
> 
> 			regards, tom lane
> 
> 




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
@ 2004-04-20 05:05 ` Frank Finner <postgresql@finner.de>
  2004-04-21 09:51   ` Re: Going from a DB using SQL_ASCII to UNICODE Peter Eisentraut <peter_e@gmx.net>
  3 siblings, 1 reply; 7+ messages in thread

From: Frank Finner @ 2004-04-20 05:05 UTC (permalink / raw)
  To: ; +Cc: pgsql-admin; lloyd@middleware-company.com; eric@middleware-company.com

Hi,

I answered to a similiar problem on the [GENERAL] list a few days ago. In
short: You can use "recode" in a pipe between dump and restore like
"pg_dump database|recode ascii..utf8|psql newdatabase" to get rid of unicode
problems while transferring from a different encoding. Please look at my
statements on the[GENERAL] list for the gory details.

Regards, Frank.


On Mon, 19 Apr 2004 15:44:33 -0400
pgsql-admin-owner@postgresql.org wrote:

> 
> Hi -
> 
> We are running PostgreSQL 7.1.4, and are finally upgrading... to 7.4.2.
> 
> While we do this upgrade, we also want to make create our DB as UNICODE (we
> are having problems with SQL_ASCII where we get a bunch of ? marks).
> 
> When we try to import the data from a SQL_ASCII dumped db, into the new
> UNICODE db, everything freaks out.
> We get errors restoring from dumps that used COPY and with INSERT.
> 
> We also tried to use the 7.4 pg_dump to make sure that it wasn't a problem
> with the old pg_dump.
> 
> The errors are: 
> 
> - ERROR:  missing data for column "noiselevel"
> - CONTEXT:  COPY messages, line 1: "4393  -1      1441    14147   0
> 2000-10-12 18:30:58-05  EJB Design Questions    \N      Hi,"
> - FATAL:  invalid frontend message type 60
> - ERROR:  invalid byte sequence for encoding "UNICODE": 0xe9616c
> 
> Anyway, if anyone has any recommendations on how to migrate from SQL_ASCII
> to UNICODE, with the knowledge that we are going from 7.1 to 7.4 at the same
> time.... You will be life-savers!
> 
> Cheers,
> 
> Dion



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Going from a DB using SQL_ASCII to UNICODE
  2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
  2004-04-20 05:05 ` Re: Going from a DB using SQL_ASCII to UNICODE Frank Finner <postgresql@finner.de>
@ 2004-04-21 09:51   ` Peter Eisentraut <peter_e@gmx.net>
  0 siblings, 0 replies; 7+ messages in thread

From: Peter Eisentraut @ 2004-04-21 09:51 UTC (permalink / raw)
  To: Frank Finner <postgresql@finner.de>; +Cc: pgsql-admin; lloyd@middleware-company.com; eric@middleware-company.com

Am Dienstag, 20. April 2004 07:05 schrieb Frank Finner:
> I answered to a similiar problem on the [GENERAL] list a few days ago. In
> short: You can use "recode" in a pipe between dump and restore like
> "pg_dump database|recode ascii..utf8|psql newdatabase" to get rid of
> unicode problems while transferring from a different encoding. Please look
> at my statements on the[GENERAL] list for the gory details.

This is not necessary.  You just need to set the client encoding correctly 
(not SQL_ASCII, but something like LATIN1).  The encoding conversion happens 
automatically, as usual.



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2004-04-21 09:51 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
2004-04-19 21:56 ` CoL <col@mportal.hu>
2004-04-19 22:02 ` CoL <col@mportal.hu>
2004-04-20 01:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-04-20 01:24   ` Dion Almaer <dion@almaer.com>
2004-04-20 05:05 ` Frank Finner <postgresql@finner.de>
2004-04-21 09:51   ` Peter Eisentraut <peter_e@gmx.net>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox