pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedUTF characters compromising data import.
4+ messages / 4 participants
[nested] [flat]
* UTF characters compromising data import.
@ 2011-02-08 10:03 Gavin Beau Baumanis <beau@palcare.com.au>
0 siblings, 2 replies; 4+ messages in thread
From: Gavin Beau Baumanis @ 2011-02-08 10:03 UTC (permalink / raw)
To: pgsql-sql
Hi Everyone,
I am trying to import some data (provided to us from an external source) from a CSV file using "\copy ...."
But I get the following error message;
invalid byte sequence for encoding "UTF8": 0xfd
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
I understand the error message - but what I don't know is what I need to set the encoding to - in order to import / use the data.
As always - thanks in advance for any help you might be able to provide.
Gavin "Beau" Baumanis
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: UTF characters compromising data import.
@ 2011-02-08 10:20 Pavel Stehule <pavel.stehule@gmail.com>
parent: Gavin Beau Baumanis <beau@palcare.com.au>
1 sibling, 0 replies; 4+ messages in thread
From: Pavel Stehule @ 2011-02-08 10:20 UTC (permalink / raw)
To: Gavin Beau Baumanis <beau@palcare.com.au>; +Cc: pgsql-sql
Hello
2011/2/8 Gavin Beau Baumanis <beau@palcare.com.au>:
> Hi Everyone,
>
> I am trying to import some data (provided to us from an external source) from a CSV file using "\copy ...."
>
> But I get the following error message;
> invalid byte sequence for encoding "UTF8": 0xfd
> HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
>
> I understand the error message - but what I don't know is what I need to set the encoding to - in order to import / use the data.
>
is impossible to import data without knowledge of encoding.
you can use a some utils, that try to select a encoding
http://linux.die.net/man/1/enca
Regards
Pavel Stehule
> As always - thanks in advance for any help you might be able to provide.
>
>
> Gavin "Beau" Baumanis
>
> --
> Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-sql
>
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: UTF characters compromising data import.
@ 2011-02-12 09:15 Jasen Betts <jasen@xnet.co.nz>
parent: Gavin Beau Baumanis <beau@palcare.com.au>
1 sibling, 1 reply; 4+ messages in thread
From: Jasen Betts @ 2011-02-12 09:15 UTC (permalink / raw)
To: pgsql-sql
On 2011-02-08, Gavin Beau Baumanis <beau@palcare.com.au> wrote:
> I understand the error message - but what I don't know is what I
> need to set the encoding to - in order to import / use the data.
if you run it through
iconv --from-code=ASCII -to-code=UTF8 -c
it'll strip out all the non-ascii symbols, without knowing the
encoding it's impossible to assign any useful meaning to them.
This step may render your data useless, it would be much better to
find out what the encoding should be.
perhaps you can figure it out by observation?
--
⚂⚃ 100% natural
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: UTF characters compromising data import.
@ 2011-02-13 11:08 Gavin Beau Baumanis <gavinb@thespidernet.com>
parent: Jasen Betts <jasen@xnet.co.nz>
0 siblings, 0 replies; 4+ messages in thread
From: Gavin Beau Baumanis @ 2011-02-13 11:08 UTC (permalink / raw)
To: Jasen Betts <jasen@xnet.co.nz>; +Cc: pgsql-sql
Hi and thanks for the replies,
I have had some luck.
I did find the encoding used originally to create the text files I am trying to import.
I have managed to use the client_encoding environmental variable and then successfully did manage to import the data.
Gavin.
On 12/02/2011, at 8:15 PM, Jasen Betts wrote:
> On 2011-02-08, Gavin Beau Baumanis <beau@palcare.com.au> wrote:
>
>> I understand the error message - but what I don't know is what I
>> need to set the encoding to - in order to import / use the data.
>
> if you run it through
>
> iconv --from-code=ASCII -to-code=UTF8 -c
>
> it'll strip out all the non-ascii symbols, without knowing the
> encoding it's impossible to assign any useful meaning to them.
> This step may render your data useless, it would be much better to
> find out what the encoding should be.
>
> perhaps you can figure it out by observation?
>
> --
> ⚂⚃ 100% natural
>
> --
> Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-sql
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2011-02-13 11:08 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-02-08 10:03 UTF characters compromising data import. Gavin Beau Baumanis <beau@palcare.com.au>
2011-02-08 10:20 ` Pavel Stehule <pavel.stehule@gmail.com>
2011-02-12 09:15 ` Jasen Betts <jasen@xnet.co.nz>
2011-02-13 11:08 ` Gavin Beau Baumanis <gavinb@thespidernet.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox