pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error
19+ messages / 14 participants
[nested] [flat]

* ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error
@ 2010-03-08 00:37 Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
  0 siblings, 1 reply; 19+ messages in thread

From: Wang, Mary Y @ 2010-03-08 00:37 UTC (permalink / raw)
  To: pgsql-general

Hi All,
I got the following error and not sure how to fix it.
"psql:/tmp/030610dumpfile.txt:4369: ERROR:  invalid byte sequence for encoding "UTF8": 0x93
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding"."

Any ideas?
I'm running on Postgres 8.3.8.

Mary





^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
@ 2010-03-08 08:17 ` Albe Laurenz <laurenz.albe@wien.gv.at>
  2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Albe Laurenz @ 2010-03-08 08:17 UTC (permalink / raw)
  To: Wang, Mary Y *EXTERN* <mary.y.wang@boeing.com>; pgsql-general

Mary Y Wang wrote:
> I got the following error and not sure how to fix it.
> "psql:/tmp/030610dumpfile.txt:4369: ERROR:  invalid byte sequence for encoding "UTF8": 0x93
> HINT:  This error can also happen if the byte sequence does 
> not match the encoding expected by the server, which is 
> controlled by "client_encoding"."
> 
> Any ideas?
> I'm running on Postgres 8.3.8.

You must either change the client encoding to the encoding of the
data you are feeding the client, or you must change the data to
match the client encoding.

Client encoding can be changed with the SQL statement
"SET client_encoding = ..." or by setting the PGCLIENTENCODING
environment variable.

Yours,
Laurenz Albe



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
@ 2010-03-08 12:38   ` Michał Pawlikowski <misiekbest@gmail.com>
  2010-03-12 20:20     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Michał Pawlikowski @ 2010-03-08 12:38 UTC (permalink / raw)
  To: pgsql-general

On Mon, Mar 8, 2010 at 9:17 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> Mary Y Wang wrote:
>> I got the following error and not sure how to fix it.
>> "psql:/tmp/030610dumpfile.txt:4369: ERROR:  invalid byte sequence for encoding "UTF8": 0x93

look here:
http://www.depesz.com/index.php/2010/03/07/error-invalid-byte-sequence-for-encoding/



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
  2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
@ 2010-03-12 20:20     ` Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-12 20:48       ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Alvaro Herrera <alvherre@commandprompt.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Wang, Mary Y @ 2010-03-12 20:20 UTC (permalink / raw)
  To: Michał Pawlikowski <misiekbest@gmail.com>; +Cc: pgsql-general

Hi Michal,

Thanks for the info.  However, the text that the user entered is all in English letters.  I saw this error, when I restored my database with the pg_dump --insert option.  That row didn't get inserted to the table.  What's interesting about this problem is that when I manually entered the data to the database via psql, this record made it without any problems. It's still a mystery to me .....

Mary


-----Original Message-----
From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] On Behalf Of Michal Pawlikowski
Sent: Monday, March 08, 2010 4:38 AM
To: pgsql-general@postgresql.org
Subject: Re: [GENERAL] ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error

On Mon, Mar 8, 2010 at 9:17 AM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> Mary Y Wang wrote:
>> I got the following error and not sure how to fix it.
>> "psql:/tmp/030610dumpfile.txt:4369: ERROR:  invalid byte sequence for 
>> encoding "UTF8": 0x93

look here:
http://www.depesz.com/index.php/2010/03/07/error-invalid-byte-sequence-for-encoding/

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
  2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
  2010-03-12 20:20     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
@ 2010-03-12 20:48       ` Alvaro Herrera <alvherre@commandprompt.com>
  2010-03-12 20:51         ` Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Alvaro Herrera @ 2010-03-12 20:48 UTC (permalink / raw)
  To: Wang, Mary Y <mary.y.wang@boeing.com>; +Cc: Michał Pawlikowski <misiekbest@gmail.com>; pgsql-general

Wang, Mary Y escribió:
> Hi Michal,
> 
> Thanks for the info.  However, the text that the user entered is all in English letters.  I saw this error, when I restored my database with the pg_dump --insert option.  That row didn't get inserted to the table.  What's interesting about this problem is that when I manually entered the data to the database via psql, this record made it without any problems. It's still a mystery to me .....

Well, 0x93 is certainly not an english letter; most likely it's a
win1252-encoded curved single quote.  That's not within the ASCII
definition.  You can probably insert the data by declaring

SET client_encoding TO 'win1252'

at the start of the file.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
  2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
  2010-03-12 20:20     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-12 20:48       ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Alvaro Herrera <alvherre@commandprompt.com>
@ 2010-03-12 20:51         ` Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-12 20:56           ` Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Wang, Mary Y @ 2010-03-12 20:51 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Michał Pawlikowski <misiekbest@gmail.com>; pgsql-general

Thanks Alvaro.  What do I do with a apostrophe  (such as I've)?

Mary 


----Original Message-----
From: Alvaro Herrera [mailto:alvherre@commandprompt.com] 
Sent: Friday, March 12, 2010 12:48 PM
To: Wang, Mary Y
Cc: Michał Pawlikowski; pgsql-general@postgresql.org
Subject: Re: [GENERAL] ERROR: invalid byte sequence for encoding "UTF8":0x93 Error

Wang, Mary Y escribió:
> Hi Michal,
> 
> Thanks for the info.  However, the text that the user entered is all in English letters.  I saw this error, when I restored my database with the pg_dump --insert option.  That row didn't get inserted to the table.  What's interesting about this problem is that when I manually entered the data to the database via psql, this record made it without any problems. It's still a mystery to me .....

Well, 0x93 is certainly not an english letter; most likely it's a win1252-encoded curved single quote.  That's not within the ASCII definition.  You can probably insert the data by declaring

SET client_encoding TO 'win1252'

at the start of the file.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error
  2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-08 08:17 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Albe Laurenz <laurenz.albe@wien.gv.at>
  2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
  2010-03-12 20:20     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
  2010-03-12 20:48       ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Alvaro Herrera <alvherre@commandprompt.com>
  2010-03-12 20:51         ` Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
@ 2010-03-12 20:56           ` Wang, Mary Y <mary.y.wang@boeing.com>
  0 siblings, 0 replies; 19+ messages in thread

From: Wang, Mary Y @ 2010-03-12 20:56 UTC (permalink / raw)
  To: Alvaro Herrera <alvherre@commandprompt.com>; +Cc: Michał Pawlikowski <misiekbest@gmail.com>; pgsql-general

 Ok. What was I thinking, the "curved single quote" that you talked about is the apostrophe.  It makes sense now.
Thanks
Mary


-----Original Message-----
From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] On Behalf Of Wang, Mary Y
Sent: Friday, March 12, 2010 12:51 PM
To: Alvaro Herrera
Cc: Michał Pawlikowski; pgsql-general@postgresql.org
Subject: Re: [GENERAL] ERROR: invalid byte sequence for encoding "UTF8":0x93 Error

Thanks Alvaro.  What do I do with a apostrophe  (such as I've)?

Mary 


----Original Message-----
From: Alvaro Herrera [mailto:alvherre@commandprompt.com]
Sent: Friday, March 12, 2010 12:48 PM
To: Wang, Mary Y
Cc: Michał Pawlikowski; pgsql-general@postgresql.org
Subject: Re: [GENERAL] ERROR: invalid byte sequence for encoding "UTF8":0x93 Error

Wang, Mary Y escribió:
> Hi Michal,
> 
> Thanks for the info.  However, the text that the user entered is all in English letters.  I saw this error, when I restored my database with the pg_dump --insert option.  That row didn't get inserted to the table.  What's interesting about this problem is that when I manually entered the data to the database via psql, this record made it without any problems. It's still a mystery to me .....

Well, 0x93 is certainly not an english letter; most likely it's a win1252-encoded curved single quote.  That's not within the ASCII definition.  You can probably insert the data by declaring

SET client_encoding TO 'win1252'

at the start of the file.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
@ 2011-02-27 12:47 AI Rumman <rummandba@gmail.com>
  2011-02-27 15:47 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Scott Ribe <scott_ribe@elevated-dev.com>
  2011-02-28 03:14 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
  0 siblings, 2 replies; 19+ messages in thread

From: AI Rumman @ 2011-02-27 12:47 UTC (permalink / raw)
  To: pgsql-general

I am getting error in Postgresql 9.0.1.

update import_details_test
set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão (Schools
21,22,76)( Ru)","http://www.asdas.aa.nj.us","","908 436
4861","","","--None--","",","","--None--","","","0","","--None--","0","2008-12-29
17:53:08","","2010-08-23 14:58:56","0","Vantage Learning-OTS","
hsmith@vantage.com","","","","","","","","","","","","","","0","","","","","","","0","","300
S BROAD
ST","","","","ELIZABETH","","NJ","","07208","","","","","UNION","0","School","0","Tier
II","","0","","","","","ELIZABETH CITY","Elementary
School","Public","500","0","0","0","Suburban","","--None--","","","","0","","","","","","","","","","","","","--None--","0","","","","","","","","","","","","","","","","0","0","","0","0","0","--None--","--None--","","","","","0","","","0","","0","","","","0","","","","","","","","","","","","","","","","","","","","","0","","","","","","","","",""]'
WHERE import_queue_id = '35' AND record_num = 181;
ERROR:  invalid byte sequence for encoding "UTF8": 0xc35c

Any idea please.

^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
  2011-02-27 12:47 ERROR: invalid byte sequence for encoding "UTF8": 0xc35c AI Rumman <rummandba@gmail.com>
@ 2011-02-27 15:47 ` Scott Ribe <scott_ribe@elevated-dev.com>
  1 sibling, 0 replies; 19+ messages in thread

From: Scott Ribe @ 2011-02-27 15:47 UTC (permalink / raw)
  To: AI Rumman <rummandba@gmail.com>; +Cc: pgsql-general

On Feb 27, 2011, at 5:47 AM, AI Rumman wrote:

> Any idea please.

Don't do that ;-)

Seriously, the error means exactly what it says, so you have to figure out why your app is trying to insert invalid UTF-8.

-- 
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice







^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
  2011-02-27 12:47 ERROR: invalid byte sequence for encoding "UTF8": 0xc35c AI Rumman <rummandba@gmail.com>
@ 2011-02-28 03:14 ` Craig Ringer <craig@postnewspapers.com.au>
  2011-03-03 01:18   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Jasmin Dizdarevic <jasmin.dizdarevic@gmail.com>
  1 sibling, 1 reply; 19+ messages in thread

From: Craig Ringer @ 2011-02-28 03:14 UTC (permalink / raw)
  To: AI Rumman <rummandba@gmail.com>; +Cc: pgsql-general

On 27/02/11 20:47, AI Rumman wrote:
> I am getting error in Postgresql 9.0.1.
> 
> update import_details_test
> set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão 
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Because your email client may have transformed the text encoding, I
can't make any certain conclusions about what you're actually sending to
the database, but it's highly likely that you're sending latin-1 encoded
text to the database while your client_encoding is set to 'utf8'.

The marked text is most likely the problem... but I think there's more
wrong with it than just being latin-1 encoded. That kind of mangling
often comes about when utf-8 text has been incorrectly interpreted as
latin-1 and modified, or when something has incorrectly tried to do
utf8<->latin-1 conversions more than once. You really need to figure out
what encoding your input is in, convert it to a known encoding like
utf-8 *once*, and keep it that way.

If you're using Python, which I suspect you might be, the "".decode()
function is useful. For example, I can convert a latin-1 encoded byte
string to a python Unicode string with:

   "somelatin1string".decode("latin-1")

Sometimes you can get away with just "SET client_encoding=latin-1" but
in this case your string data looks like it's been mangled by more than
just a single encoding mis-interpretation, so you'll probably just
silently insert corrupt data by doing that. Don't. Fix your code so it
knows what the text encoding of the input is.

If you are, in fact, using Python, it's a really good idea to always
"".decode() all your inputs so your internal processing is done in
Unicode (UTF-16, in fact). Similarly, Qt programmers should convert
everything to unicode QString as soon as possible and use that for all
internal manipulation. It'll save a lot of pain.




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
  2011-02-27 12:47 ERROR: invalid byte sequence for encoding "UTF8": 0xc35c AI Rumman <rummandba@gmail.com>
  2011-02-28 03:14 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
@ 2011-03-03 01:18   ` Jasmin Dizdarevic <jasmin.dizdarevic@gmail.com>
  2011-03-03 01:45     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
  0 siblings, 1 reply; 19+ messages in thread

From: Jasmin Dizdarevic @ 2011-03-03 01:18 UTC (permalink / raw)
  To: Craig Ringer <craig@postnewspapers.com.au>; pgsql-general

@ALL: Isn't it possible and wise to include an (optional) encoder in pgsql?

we're importing a lot of data from textfiles, which are not utf-8. we always
have to change the encoding in another tool before using COPY.



2011/2/28 Craig Ringer <craig@postnewspapers.com.au>

> On 27/02/11 20:47, AI Rumman wrote:
> > I am getting error in Postgresql 9.0.1.
> >
> > update import_details_test
> > set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão
>                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Because your email client may have transformed the text encoding, I
> can't make any certain conclusions about what you're actually sending to
> the database, but it's highly likely that you're sending latin-1 encoded
> text to the database while your client_encoding is set to 'utf8'.
>
> The marked text is most likely the problem... but I think there's more
> wrong with it than just being latin-1 encoded. That kind of mangling
> often comes about when utf-8 text has been incorrectly interpreted as
> latin-1 and modified, or when something has incorrectly tried to do
> utf8<->latin-1 conversions more than once. You really need to figure out
> what encoding your input is in, convert it to a known encoding like
> utf-8 *once*, and keep it that way.
>
> If you're using Python, which I suspect you might be, the "".decode()
> function is useful. For example, I can convert a latin-1 encoded byte
> string to a python Unicode string with:
>
>   "somelatin1string".decode("latin-1")
>
> Sometimes you can get away with just "SET client_encoding=latin-1" but
> in this case your string data looks like it's been mangled by more than
> just a single encoding mis-interpretation, so you'll probably just
> silently insert corrupt data by doing that. Don't. Fix your code so it
> knows what the text encoding of the input is.
>
> If you are, in fact, using Python, it's a really good idea to always
> "".decode() all your inputs so your internal processing is done in
> Unicode (UTF-16, in fact). Similarly, Qt programmers should convert
> everything to unicode QString as soon as possible and use that for all
> internal manipulation. It'll save a lot of pain.
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c
  2011-02-27 12:47 ERROR: invalid byte sequence for encoding "UTF8": 0xc35c AI Rumman <rummandba@gmail.com>
  2011-02-28 03:14 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
  2011-03-03 01:18   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Jasmin Dizdarevic <jasmin.dizdarevic@gmail.com>
@ 2011-03-03 01:45     ` Craig Ringer <craig@postnewspapers.com.au>
  0 siblings, 0 replies; 19+ messages in thread

From: Craig Ringer @ 2011-03-03 01:45 UTC (permalink / raw)
  To: Jasmin Dizdarevic <jasmin.dizdarevic@gmail.com>; +Cc: pgsql-general

On 03/03/11 09:18, Jasmin Dizdarevic wrote:
> @ALL: Isn't it possible and wise to include an (optional) encoder in pgsql?
> 
> we're importing a lot of data from textfiles, which are not utf-8. we
> always have to change the encoding in another tool before using COPY. 

SET client_encoding

--
Craig Ringer



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* ERROR:  invalid byte sequence for encoding "UTF8": 0xc325
@ 2012-03-30 17:00 Prashant Bharucha <prashantbharucha@yahoo.ca>
  2012-03-30 17:44 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xc325 Alan Hodgson <ahodgson@simkin.ca>
  0 siblings, 1 reply; 19+ messages in thread

From: Prashant Bharucha @ 2012-03-30 17:00 UTC (permalink / raw)
  To: pgsql-general

Hello All

Could you help me to automatically convert all db request into UTF8 encode ?

Thx
Prashant

^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xc325
  2012-03-30 17:00 ERROR:  invalid byte sequence for encoding "UTF8": 0xc325 Prashant Bharucha <prashantbharucha@yahoo.ca>
@ 2012-03-30 17:44 ` Alan Hodgson <ahodgson@simkin.ca>
  0 siblings, 0 replies; 19+ messages in thread

From: Alan Hodgson @ 2012-03-30 17:44 UTC (permalink / raw)
  To: pgsql-general

On Friday, March 30, 2012 10:00:31 AM Prashant Bharucha wrote:
> Hello All
> 
> Could you help me to automatically convert all db request into UTF8 encode ?
> 

Set your session client_encoding to match your data. That's about as close as 
you can get to automatic.

  http://www.postgresql.org/docs/9.1/static/multibyte.html

A better bet is to make your application end-to-end UTF8, which is doable, 
more or less, in a web environment, although misbehaving clients will still 
sometimes send you bad data. For any other data source (especially email) 
you'll probably get tons of badly encoded data.

If you're looking for a silver bullet, there isn't one.




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* ERROR:  invalid byte sequence for encoding "UTF8": 0xfc
@ 2018-03-27 20:47 armand pirvu <armand.pirvu@gmail.com>
  2018-03-27 22:30 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc armand pirvu <armand.pirvu@gmail.com>
  0 siblings, 1 reply; 19+ messages in thread

From: armand pirvu @ 2018-03-27 20:47 UTC (permalink / raw)
  To: pgsql-general

Folks


From another system (non postgres) as data gets changed we collect in the form like below into a named pipe which is tailed into psql -U csidba -d repdb -hrephost

insert into foo (mydata) values ('{"mail_stop": "" , "city_id": "Garching bei M?nchen"}');

And got a failure 
ERROR:  invalid byte sequence for encoding "UTF8": 0xfc


insert into foo (mydata) values ('{"mail_stop": "" , "city_id": "Garching bei M?nchen"}');

Problem is said system to convert to UTF8 is nicely put a huge amount of effort and we don't have the resources anyway at this point


This is not a copy, just a tailed named pipe into psql

Is there any trick/way I can shove this data into Postgres ? I really don't care whether it read "Garching bei M?nchen" or "Garching bei Mfoonchen"



Thanks
— Armand






^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc
  2018-03-27 20:47 ERROR:  invalid byte sequence for encoding "UTF8": 0xfc armand pirvu <armand.pirvu@gmail.com>
@ 2018-03-27 22:30 ` armand pirvu <armand.pirvu@gmail.com>
  0 siblings, 0 replies; 19+ messages in thread

From: armand pirvu @ 2018-03-27 22:30 UTC (permalink / raw)
  To: pgsql-general


> On Mar 27, 2018, at 3:47 PM, armand pirvu <armand.pirvu@gmail.com> wrote:
> 
> Folks
> 
> 
> From another system (non postgres) as data gets changed we collect in the form like below into a named pipe which is tailed into psql -U csidba -d repdb -hrephost
> 
> insert into foo (mydata) values ('{"mail_stop": "" , "city_id": "Garching bei M?nchen"}');
> 
> And got a failure 
> ERROR:  invalid byte sequence for encoding "UTF8": 0xfc
> 
> 
> insert into foo (mydata) values ('{"mail_stop": "" , "city_id": "Garching bei M?nchen"}');
> 
> Problem is said system to convert to UTF8 is nicely put a huge amount of effort and we don't have the resources anyway at this point
> 
> 
> This is not a copy, just a tailed named pipe into psql
> 
> Is there any trick/way I can shove this data into Postgres ? I really don't care whether it read "Garching bei M?nchen" or "Garching bei Mfoonchen"
> 
> 
> 
> Thanks
> — Armand
> 
> 

Sorry for double posting but I found that if I change the client_encoding to WIN1252 I am able to move forward

"city_id": "Garching bei München”



 



^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* ERROR: invalid byte sequence for encoding UTF8: 0x00
@ 2024-01-09 12:18 Kiran K V <kirankv.1982@gmail.com>
  2024-01-09 15:13 ` Re: ERROR: invalid byte sequence for encoding UTF8: 0x00 Laurenz Albe <laurenz.albe@cybertec.at>
  0 siblings, 1 reply; 19+ messages in thread

From: Kiran K V @ 2024-01-09 12:18 UTC (permalink / raw)
  To: pgsql-general@lists.postgresql.org

Hi,

I have a table with bytea column and trying to load the data using copy
command.  But Copy command is failing with "ERROR: invalid byte sequence
for encoding UTF8: 0x00. Why postgresql is failing to load when data
contains 0x00. How to resolve this error ? any workaround to load the data
with 0x00 values ?

Please advise. Thank you very much.

Regards,
Kiran

^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding UTF8: 0x00
  2024-01-09 12:18 ERROR: invalid byte sequence for encoding UTF8: 0x00 Kiran K V <kirankv.1982@gmail.com>
@ 2024-01-09 15:13 ` Laurenz Albe <laurenz.albe@cybertec.at>
  2024-01-09 15:48   ` Re: ERROR: invalid byte sequence for encoding UTF8: 0x00 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 19+ messages in thread

From: Laurenz Albe @ 2024-01-09 15:13 UTC (permalink / raw)
  To: Kiran K V <kirankv.1982@gmail.com>; pgsql-general@lists.postgresql.org

On Tue, 2024-01-09 at 17:48 +0530, Kiran K V wrote:
> I have a table with bytea column and trying to load the data using copy command.
> But Copy command is failing with
> "ERROR: invalid byte sequence for encoding UTF8: 0x00.
> Why postgresql is failing to load when data contains 0x00.
> How to resolve this error ? any workaround to load the data with 0x00 values ?

This is not about loading data into a "bytea", but into a "text", "varchar" or
"char" column.  Check again.

Yours,
Laurenz Albe





^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: ERROR: invalid byte sequence for encoding UTF8: 0x00
  2024-01-09 12:18 ERROR: invalid byte sequence for encoding UTF8: 0x00 Kiran K V <kirankv.1982@gmail.com>
  2024-01-09 15:13 ` Re: ERROR: invalid byte sequence for encoding UTF8: 0x00 Laurenz Albe <laurenz.albe@cybertec.at>
@ 2024-01-09 15:48   ` Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 19+ messages in thread

From: Tom Lane @ 2024-01-09 15:48 UTC (permalink / raw)
  To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Kiran K V <kirankv.1982@gmail.com>; pgsql-general@lists.postgresql.org

Laurenz Albe <laurenz.albe@cybertec.at> writes:
> On Tue, 2024-01-09 at 17:48 +0530, Kiran K V wrote:
>> I have a table with bytea column and trying to load the data using copy command.
>> But Copy command is failing with
>> "ERROR: invalid byte sequence for encoding UTF8: 0x00.
>> Why postgresql is failing to load when data contains 0x00.
>> How to resolve this error ? any workaround to load the data with 0x00 values ?

> This is not about loading data into a "bytea", but into a "text", "varchar" or
> "char" column.  Check again.

I am guessing that the OP tried to put a literal NUL character
into the COPY data.  A bytea value in COPY data needs to be
text-encoded as per one of the alternatives explained in the
manual.  Alternatively, you could jump through the hoops involved
in sending binary COPY data.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 19+ messages in thread


end of thread, other threads:[~2024-01-09 15:48 UTC | newest]

Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-03-08 00:37 ERROR:  invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
2010-03-08 08:17 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2010-03-08 12:38   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Michał Pawlikowski <misiekbest@gmail.com>
2010-03-12 20:20     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
2010-03-12 20:48       ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0x93 Error Alvaro Herrera <alvherre@commandprompt.com>
2010-03-12 20:51         ` Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
2010-03-12 20:56           ` Re: ERROR: invalid byte sequence for encoding "UTF8":0x93 Error Wang, Mary Y <mary.y.wang@boeing.com>
2011-02-27 12:47 ERROR: invalid byte sequence for encoding "UTF8": 0xc35c AI Rumman <rummandba@gmail.com>
2011-02-27 15:47 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Scott Ribe <scott_ribe@elevated-dev.com>
2011-02-28 03:14 ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
2011-03-03 01:18   ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Jasmin Dizdarevic <jasmin.dizdarevic@gmail.com>
2011-03-03 01:45     ` Re: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c Craig Ringer <craig@postnewspapers.com.au>
2012-03-30 17:00 ERROR:  invalid byte sequence for encoding "UTF8": 0xc325 Prashant Bharucha <prashantbharucha@yahoo.ca>
2012-03-30 17:44 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xc325 Alan Hodgson <ahodgson@simkin.ca>
2018-03-27 20:47 ERROR:  invalid byte sequence for encoding "UTF8": 0xfc armand pirvu <armand.pirvu@gmail.com>
2018-03-27 22:30 ` Re: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc armand pirvu <armand.pirvu@gmail.com>
2024-01-09 12:18 ERROR: invalid byte sequence for encoding UTF8: 0x00 Kiran K V <kirankv.1982@gmail.com>
2024-01-09 15:13 ` Re: ERROR: invalid byte sequence for encoding UTF8: 0x00 Laurenz Albe <laurenz.albe@cybertec.at>
2024-01-09 15:48   ` Re: ERROR: invalid byte sequence for encoding UTF8: 0x00 Tom Lane <tgl@sss.pgh.pa.us>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox