pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedGoing from a DB using SQL_ASCII to UNICODE
7+ messages / 5 participants
[nested] [flat]
* Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-19 19:44 Dion Almaer <dion@almaer.com>
0 siblings, 4 replies; 7+ messages in thread
From: Dion Almaer @ 2004-04-19 19:44 UTC (permalink / raw)
To: pgsql-admin; +Cc: 'Lloyd Benson' <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>
Hi -
We are running PostgreSQL 7.1.4, and are finally upgrading... to 7.4.2.
While we do this upgrade, we also want to make create our DB as UNICODE (we
are having problems with SQL_ASCII where we get a bunch of ? marks).
When we try to import the data from a SQL_ASCII dumped db, into the new
UNICODE db, everything freaks out.
We get errors restoring from dumps that used COPY and with INSERT.
We also tried to use the 7.4 pg_dump to make sure that it wasn't a problem
with the old pg_dump.
The errors are:
- ERROR: missing data for column "noiselevel"
- CONTEXT: COPY messages, line 1: "4393 -1 1441 14147 0
2000-10-12 18:30:58-05 EJB Design Questions \N Hi,"
- FATAL: invalid frontend message type 60
- ERROR: invalid byte sequence for encoding "UNICODE": 0xe9616c
Anyway, if anyone has any recommendations on how to migrate from SQL_ASCII
to UNICODE, with the knowledge that we are going from 7.1 to 7.4 at the same
time.... You will be life-savers!
Cheers,
Dion
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-19 21:56 CoL <col@mportal.hu>
parent: Dion Almaer <dion@almaer.com>
3 siblings, 0 replies; 7+ messages in thread
From: CoL @ 2004-04-19 21:56 UTC (permalink / raw)
To: pgsql-admin
hi,
Dion Almaer wrote, On 4/19/2004 21:44:
> - ERROR: missing data for column "noiselevel"
> - CONTEXT: COPY messages, line 1: "4393 -1 1441 14147 0
> 2000-10-12 18:30:58-05 EJB Design Questions \N Hi,"
for this one: better to dump with: -d
C.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-19 22:02 CoL <col@mportal.hu>
parent: Dion Almaer <dion@almaer.com>
3 siblings, 0 replies; 7+ messages in thread
From: CoL @ 2004-04-19 22:02 UTC (permalink / raw)
To: pgsql-admin
hi,
Dion Almaer wrote, On 4/19/2004 21:44:
> - ERROR: invalid byte sequence for encoding "UNICODE": 0xe9616c
and for this: convert your dump with iconv to unicode, then load it.
C.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-20 01:20 Tom Lane <tgl@sss.pgh.pa.us>
parent: Dion Almaer <dion@almaer.com>
3 siblings, 1 reply; 7+ messages in thread
From: Tom Lane @ 2004-04-20 01:20 UTC (permalink / raw)
To: Dion Almaer <dion@almaer.com>; +Cc: pgsql-admin@postgresql.org, "'Lloyd Benson'" <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>
"Dion Almaer" <dion@almaer.com> writes:
> When we try to import the data from a SQL_ASCII dumped db, into the new
> UNICODE db, everything freaks out.
I'm no expert on this stuff, but I think what you need to do is add
set client_encoding = sql_ascii;
to the top of the dump file. (As of fairly recently, pg_dump will
automatically add such a SET, but I'm pretty sure 7.4.2 won't.)
If that doesn't help, then what you have is actually not valid UTF-8
data, and what you'll have to do is figure out what encoding it's in and
specify that instead. If it's in a mishmash of different encodings,
you're in for some pain :-(
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-20 01:24 Dion Almaer <dion@almaer.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 7+ messages in thread
From: Dion Almaer @ 2004-04-20 01:24 UTC (permalink / raw)
To: 'Tom Lane' <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin; Lloyd Benson <lloyd@middleware-company.com>; Eric Preston <eric@middleware-company.com>
Tom,
I will try to add "set client_encoding = sql_ascii;" and pray :)
Cheers,
Dion
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Monday, April 19, 2004 9:21 PM
> To: Dion Almaer
> Cc: pgsql-admin@postgresql.org; 'Lloyd Benson'; Eric Preston
> Subject: Re: [ADMIN] Going from a DB using SQL_ASCII to UNICODE
>
> "Dion Almaer" <dion@almaer.com> writes:
> > When we try to import the data from a SQL_ASCII dumped db, into the
> > new UNICODE db, everything freaks out.
>
> I'm no expert on this stuff, but I think what you need to do is add
>
> set client_encoding = sql_ascii;
>
> to the top of the dump file. (As of fairly recently, pg_dump
> will automatically add such a SET, but I'm pretty sure 7.4.2 won't.)
>
> If that doesn't help, then what you have is actually not
> valid UTF-8 data, and what you'll have to do is figure out
> what encoding it's in and specify that instead. If it's in a
> mishmash of different encodings, you're in for some pain :-(
>
> regards, tom lane
>
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-20 05:05 Frank Finner <postgresql@finner.de>
parent: Dion Almaer <dion@almaer.com>
3 siblings, 1 reply; 7+ messages in thread
From: Frank Finner @ 2004-04-20 05:05 UTC (permalink / raw)
To: ; +Cc: pgsql-admin; lloyd@middleware-company.com; eric@middleware-company.com
Hi,
I answered to a similiar problem on the [GENERAL] list a few days ago. In
short: You can use "recode" in a pipe between dump and restore like
"pg_dump database|recode ascii..utf8|psql newdatabase" to get rid of unicode
problems while transferring from a different encoding. Please look at my
statements on the[GENERAL] list for the gory details.
Regards, Frank.
On Mon, 19 Apr 2004 15:44:33 -0400
pgsql-admin-owner@postgresql.org wrote:
>
> Hi -
>
> We are running PostgreSQL 7.1.4, and are finally upgrading... to 7.4.2.
>
> While we do this upgrade, we also want to make create our DB as UNICODE (we
> are having problems with SQL_ASCII where we get a bunch of ? marks).
>
> When we try to import the data from a SQL_ASCII dumped db, into the new
> UNICODE db, everything freaks out.
> We get errors restoring from dumps that used COPY and with INSERT.
>
> We also tried to use the 7.4 pg_dump to make sure that it wasn't a problem
> with the old pg_dump.
>
> The errors are:
>
> - ERROR: missing data for column "noiselevel"
> - CONTEXT: COPY messages, line 1: "4393 -1 1441 14147 0
> 2000-10-12 18:30:58-05 EJB Design Questions \N Hi,"
> - FATAL: invalid frontend message type 60
> - ERROR: invalid byte sequence for encoding "UNICODE": 0xe9616c
>
> Anyway, if anyone has any recommendations on how to migrate from SQL_ASCII
> to UNICODE, with the knowledge that we are going from 7.1 to 7.4 at the same
> time.... You will be life-savers!
>
> Cheers,
>
> Dion
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Going from a DB using SQL_ASCII to UNICODE
@ 2004-04-21 09:51 Peter Eisentraut <peter_e@gmx.net>
parent: Frank Finner <postgresql@finner.de>
0 siblings, 0 replies; 7+ messages in thread
From: Peter Eisentraut @ 2004-04-21 09:51 UTC (permalink / raw)
To: Frank Finner <postgresql@finner.de>; +Cc: pgsql-admin; lloyd@middleware-company.com; eric@middleware-company.com
Am Dienstag, 20. April 2004 07:05 schrieb Frank Finner:
> I answered to a similiar problem on the [GENERAL] list a few days ago. In
> short: You can use "recode" in a pipe between dump and restore like
> "pg_dump database|recode ascii..utf8|psql newdatabase" to get rid of
> unicode problems while transferring from a different encoding. Please look
> at my statements on the[GENERAL] list for the gory details.
This is not necessary. You just need to set the client encoding correctly
(not SQL_ASCII, but something like LATIN1). The encoding conversion happens
automatically, as usual.
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2004-04-21 09:51 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-04-19 19:44 Going from a DB using SQL_ASCII to UNICODE Dion Almaer <dion@almaer.com>
2004-04-19 21:56 ` CoL <col@mportal.hu>
2004-04-19 22:02 ` CoL <col@mportal.hu>
2004-04-20 01:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-04-20 01:24 ` Dion Almaer <dion@almaer.com>
2004-04-20 05:05 ` Frank Finner <postgresql@finner.de>
2004-04-21 09:51 ` Peter Eisentraut <peter_e@gmx.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox