From: Craig Ringer <craig@postnewspapers.com.au>
To: AI Rumman <rummandba@gmail.com>
Cc: pgsql-general General <pgsql-general@postgresql.org>
Subject: Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
Date: Mon, 28 Feb 2011 11:14:13 +0800
Message-ID: <4D6B1305.5010400@postnewspapers.com.au> (raw)
In-Reply-To: <AANLkTikfs1JYDtY1bcQFf+-oTs8Tmx89wse8V5Lp+xsJ@mail.gmail.com>
References: <AANLkTikfs1JYDtY1bcQFf+-oTs8Tmx89wse8V5Lp+xsJ@mail.gmail.com>
On 27/02/11 20:47, AI Rumman wrote:
> I am getting error in Postgresql 9.0.1.
>
> update import_details_test
> set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Because your email client may have transformed the text encoding, I
can't make any certain conclusions about what you're actually sending to
the database, but it's highly likely that you're sending latin-1 encoded
text to the database while your client_encoding is set to 'utf8'.
The marked text is most likely the problem... but I think there's more
wrong with it than just being latin-1 encoded. That kind of mangling
often comes about when utf-8 text has been incorrectly interpreted as
latin-1 and modified, or when something has incorrectly tried to do
utf8<->latin-1 conversions more than once. You really need to figure out
what encoding your input is in, convert it to a known encoding like
utf-8 *once*, and keep it that way.
If you're using Python, which I suspect you might be, the "".decode()
function is useful. For example, I can convert a latin-1 encoded byte
string to a python Unicode string with:
"somelatin1string".decode("latin-1")
Sometimes you can get away with just "SET client_encoding=latin-1" but
in this case your string data looks like it's been mangled by more than
just a single encoding mis-interpretation, so you'll probably just
silently insert corrupt data by doing that. Don't. Fix your code so it
knows what the text encoding of the input is.
If you are, in fact, using Python, it's a really good idea to always
"".decode() all your inputs so your internal processing is done in
Unicode (UTF-16, in fact). Similarly, Qt programmers should convert
everything to unicode QString as soon as possible and use that for all
internal manipulation. It'll save a lot of pain.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: craig@postnewspapers.com.au, rummandba@gmail.com
Subject: Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
In-Reply-To: <4D6B1305.5010400@postnewspapers.com.au>
References: <AANLkTikfs1JYDtY1bcQFf+-oTs8Tmx89wse8V5Lp+xsJ@mail.gmail.com> <4D6B1305.5010400@postnewspapers.com.au>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox