pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
UNICODE
23+ messages / 10 participants
[nested] [flat]

* UNICODE
@ 2001-10-28 08:22  Jean-Michel POURE <jm.poure@freesurf.fr>
  0 siblings, 5 replies; 23+ messages in thread

From: Jean-Michel POURE @ 2001-10-28 08:22 UTC (permalink / raw)
  To: pgsql-hackers@postgresql.org

Dear all,

I am running PostgreSQL 7.1.2 with UNICODE support in production.
Maybe I miss something about UNICODE:

CREATE TABLE "test" (
   "source_oid" serial,
   "source_timestamp" timestamp,
   "source_creation" date DEFAULT 'now',
   "source_modification" date DEFAULT 'now',
   "source_content" text
);

INSERT INTO test (source_content) VALUES ('Photocopie du permis de 
construire accepté.');

Now, when trying :
SELECT * FROM test WHERE source_content ILIKE '%accept%';  ---> returns the 
record;
SELECT * FROM test WHERE source_content ILIKE '%accepté%' ---> returns nothing
SELECT * FROM test WHERE source_content ILIKE '%accepte%' ---> returns nothing

The same happens from ODBC, PHP and psql. Can you reproduce this?

I have tried
Best regards,
Jean-Michel POURE



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 09:50  Marko Kreen <marko@l-t.ee>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  4 siblings, 0 replies; 23+ messages in thread

From: Marko Kreen @ 2001-10-28 09:50 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: pgsql-hackers@postgresql.org

On Sun, Oct 28, 2001 at 09:22:24AM +0100, Jean-Michel POURE wrote:
> Dear all,
> 
> I am running PostgreSQL 7.1.2 with UNICODE support in production.
> Maybe I miss something about UNICODE:

> SELECT * FROM test WHERE source_content ILIKE '%accepté%' ---> returns 
> nothing
> SELECT * FROM test WHERE source_content ILIKE '%accepte%' ---> returns 
> nothing
> 
> The same happens from ODBC, PHP and psql. Can you reproduce this?

1) Did you compile PostgreSQL with --enable-locale
2) Did you set correct locale for postmaster (LANG=xxx)

-- 
marko




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 10:06  Jean-Michel POURE <jm.poure@freesurf.fr>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  4 siblings, 0 replies; 23+ messages in thread

From: Jean-Michel POURE @ 2001-10-28 10:06 UTC (permalink / raw)
  To: Marko Kreen <marko@l-t.ee>; +Cc: pgsql-hackers@postgresql.org


>1) Did you compile PostgreSQL with --enable-locale
Yes.

>2) Did you set correct locale for postmaster (LANG=xxx)
Database was create using CREATE db WITH ENCODING='UNICODE'.
pgsql: \encoding returns UNICODE.

The db stores multiple languages (French, English, Japanese).
Why should I define a *single* locale for postmaster?
Do I miss something?

Best regards,
Jean-Michel POURE




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 10:53  Marko Kreen <marko@l-t.ee>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  4 siblings, 0 replies; 23+ messages in thread

From: Marko Kreen @ 2001-10-28 10:53 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: pgsql-hackers@postgresql.org

On Sun, Oct 28, 2001 at 09:22:24AM +0100, Jean-Michel POURE wrote:
> 
> I am running PostgreSQL 7.1.2 with UNICODE support in production.
> Maybe I miss something about UNICODE:
> 
> CREATE TABLE "test" (
>   "source_oid" serial,
>   "source_timestamp" timestamp,
>   "source_creation" date DEFAULT 'now',
>   "source_modification" date DEFAULT 'now',
>   "source_content" text
> );
> 
> INSERT INTO test (source_content) VALUES ('Photocopie du permis de 
> construire accepté.');
> 
> Now, when trying :
> SELECT * FROM test WHERE source_content ILIKE '%accept%';  ---> returns the 
> record;
> SELECT * FROM test WHERE source_content ILIKE '%accepté%' ---> returns 
> nothing
> SELECT * FROM test WHERE source_content ILIKE '%accepte%' ---> returns 
> nothing
> 
> The same happens from ODBC, PHP and psql. Can you reproduce this?

Sorry, I misinterpreted what your problem is.  I somehow thought
you want the 'é' and 'e' produce same result - for that you need
to mess with locale, but LIKE does not use locale anyway...

Now I reread you message and here's hint:

* If client_encoding == server_encoding, the bytes are put into
  DB as-is - no conversion is done.

So are you abslutely sure you have on client side UTF8 strings?
Unfortunately you cant use client_encoding=latin1 as PostgreSQL 
refuses the do the conversion between them.  (I am with 7.1.3)

Eg. I did the following:

* created db with encoding = UNICODE
* Put your example into test.sql
* iconv -f latin1 -t utf8 test.sql > test2.sql
* psql < test2.sql

and it worked as it should...


-- 
marko




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 11:44  Jean-Michel POURE <jm.poure@freesurf.fr>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  4 siblings, 2 replies; 23+ messages in thread

From: Jean-Michel POURE @ 2001-10-28 11:44 UTC (permalink / raw)
  To: Marko Kreen <marko@l-t.ee>; +Cc: pgsql-hackers@postgresql.org

I only want this query to work under Unicode:
SELECT * FROM test WHERE source_content ILIKE '%accepté%'.

>* If client_encoding == server_encoding, the bytes are put into
>   DB as-is - no conversion is done.
>
>So are you absolutely sure you have on client side UTF8 strings?
PostgreSQL is compiled with UNICODE and LOCALE support.
Unicode is used on both ends (PostgreSQL and psql).

>Unfortunately you cant use client_encoding=latin1 as PostgreSQL
>refuses the do the conversion between them.  (I am with 7.1.3)
According to the on-line manual, only MULE provides instant transcoding.

>Eg. I did the following:
>
>* created db with encoding = UNICODE
>* Put your example into test.sql
>* iconv -f latin1 -t utf8 test.sql > test2.sql
>* psql < test2.sql
>
>and it worked as it should...

Nice to hear it works when transcoding files to UTF-8. It shows it is not a 
back-end problem.

As for me, I typed INSERT INTO source_content VALUES ('Permis de conduire 
accepté') in Psql.
Psql does not insert the data and I have to kill it manually. Can you 
reproduce this?

Best regards,
Jean-Michel POURE



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 12:05  Marko Kreen <marko@l-t.ee>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  1 sibling, 0 replies; 23+ messages in thread

From: Marko Kreen @ 2001-10-28 12:05 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: pgsql-hackers@postgresql.org

On Sun, Oct 28, 2001 at 12:44:26PM +0100, Jean-Michel POURE wrote:
> I only want this query to work under Unicode:
> SELECT * FROM test WHERE source_content ILIKE '%accepté%'.

As I showed it works, if data in db is in UTF-8 and the query
string 'accepté' is in UTF8

> >* If client_encoding == server_encoding, the bytes are put into
> >  DB as-is - no conversion is done.
> >
> >So are you absolutely sure you have on client side UTF8 strings?
> PostgreSQL is compiled with UNICODE and LOCALE support.
> Unicode is used on both ends (PostgreSQL and psql).

psql uses your input literally - so is your console/xterm in
UNICODE/UTF8?

> >Eg. I did the following:
> >
> >* created db with encoding = UNICODE
> >* Put your example into test.sql
> >* iconv -f latin1 -t utf8 test.sql > test2.sql
> >* psql < test2.sql
> >
> >and it worked as it should...
> 
> Nice to hear it works when transcoding files to UTF-8. It shows it is not a 
> back-end problem.
> 
> As for me, I typed INSERT INTO source_content VALUES ('Permis de conduire 
> accepté') in Psql.

As I said - psql does not do any conversion.

> Psql does not insert the data and I have to kill it manually. Can you 
> reproduce this?

No.  If it hangs this is serious problem.  Or did you simply
forgot final ';' ?   It btw does not seem valid sql to me,
considering you previously provided table structure.

In the end: are the strings/queries you give to psql/pg_exec
UTF-8 - this is now main thing, as you have _configured_
everything correctly.

-- 
marko




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 13:34  Jean-Michel POURE <jm.poure@freesurf.fr>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  1 sibling, 3 replies; 23+ messages in thread

From: Jean-Michel POURE @ 2001-10-28 13:34 UTC (permalink / raw)
  To: Marko Kreen <marko@l-t.ee>; +Cc: pgsql-hackers@postgresql.org


>psql uses your input literally - so is your console/xterm in
>UNICODE/UTF8?
Client: \encoding returns 'UNICODE'.
Server: \list show databases. All databases are UNICODE (except TEMPLATE0 
and TEMPLATE1 which are ASCII of course). I use a Mandrake 8.1 distribution 
and think my console is UNICODE.

> > As for me, I typed INSERT INTO source_content VALUES ('Permis de conduire
> > accepté') in Psql.
>As I said - psql does not do any conversion.
The faulty query is: INSERT INTO test (source_content) VALUES ('Permis de 
conduire accepté');

I just can't believe that Psql is not UTF-8 compatible. It seems unreal as 
Psql is PostgreSQL #1 helper application. Should I use PostgreSQL MULE 
encoding to have automatic trans coding. What are the guidelines, I am 
completely lost.

> > Psql does not insert the data and I have to kill it manually. Can you
> > reproduce this?
>No.  If it hangs this is serious problem.  Or did you simply
>forgot final ';' ?   It btw does not seem valid sql to me,
>considering you previously provided table structure.
Is it possible that my database is corrupted? I have used pg_dump several 
times to dump data from production server to development servers and 
conversely. Does pg_dump produce UTF8 output? What are the guidelines when 
using UTF-8: forget psql and pg_dump?

>In the end: are the strings/queries you give to psql/pg_exec
>UTF-8 - this is now main thing, as you have _configured_
>everything correctly.
Everything is configured correctly server-side (PostgreSQL, Psql).

Thank you very much for your support Marko,
Best regards,
Jean-Michel





^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 15:09  Marko Kreen <marko@l-t.ee>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  2 siblings, 0 replies; 23+ messages in thread

From: Marko Kreen @ 2001-10-28 15:09 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: pgsql-hackers@postgresql.org

On Sun, Oct 28, 2001 at 02:34:49PM +0100, Jean-Michel POURE wrote:
> 
> >psql uses your input literally - so is your console/xterm in
> >UNICODE/UTF8?
> Client: \encoding returns 'UNICODE'.
> Server: \list show databases. All databases are UNICODE (except TEMPLATE0 
> and TEMPLATE1 which are ASCII of course). I use a Mandrake 8.1 distribution 
> and think my console is UNICODE.

You think?  Try this:

	$ echo "accepté" | od -c

If your term is in utf you should get:

	0000000   a   c   c   e   p   t 303 251  \n
	0000011

If in iso-8859-1:

	0000000   a   c   c   e   p   t 351  \n
	0000010

It may be in some other 8bit encoding too, then the last number
may be different.

> >> As for me, I typed INSERT INTO source_content VALUES ('Permis de conduire
> >> accepté') in Psql.
> >As I said - psql does not do any conversion.
> The faulty query is: INSERT INTO test (source_content) VALUES ('Permis de 
> conduire accepté');

Hmm.  It may be a bug in input routines.  You give PostgreSQL a
1byte 'é', it expects 2 byte char and overflows somewhere.  Can
you reproduce it on 7.1.3?  Maybe its fixed there, I cant
reproduce it.

> I just can't believe that Psql is not UTF-8 compatible. It seems unreal as 
> Psql is PostgreSQL #1 helper application. Should I use PostgreSQL MULE 
> encoding to have automatic trans coding. What are the guidelines, I am 
> completely lost.

psql & pg_dump are fine.  Your problem is that you dont give to
psql and pg_exec/PHP utf-8 strings, but some iso-8859-*.

> >> Psql does not insert the data and I have to kill it manually. Can you
> >> reproduce this?
> >No.  If it hangs this is serious problem.  Or did you simply
> >forgot final ';' ?   It btw does not seem valid sql to me,
> >considering you previously provided table structure.
> Is it possible that my database is corrupted? I have used pg_dump several 
> times to dump data from production server to development servers and 
> conversely. Does pg_dump produce UTF8 output? What are the guidelines when 
> using UTF-8: forget psql and pg_dump?

As I said, psql & pg_dump are fine, they do not touch your data
when it passes through them.

It may be that all of your database is in latin1, as you
inserted strings in this encoding, not utf8.  Basically
PostgreSQL server also does not touch your data, only its
compare functions does not work, as the strings are not in
encoding you tell they are.

Solution to this is to dump your data, use the iconv utility
to convert it to utf8 and reload.


To see this you should do:

	$ psql -c "SELECT source_contect FROM table where ..." \
		| od -c

And then look whether the weird characters are represented in
1 or 2 bytes.

-- 
marko




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 15:37  Jean-Michel POURE <jm.poure@freesurf.fr>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  2 siblings, 1 reply; 23+ messages in thread

From: Jean-Michel POURE @ 2001-10-28 15:37 UTC (permalink / raw)
  To: Marko Kreen <marko@l-t.ee>; +Cc: pgsql-hackers@postgresql.org

At 17:09 28/10/01 +0200, you wrote:
>On Sun, Oct 28, 2001 at 02:34:49PM +0100, Jean-Michel POURE wrote:
> >
> > >psql uses your input literally - so is your console/xterm in
> > >UNICODE/UTF8?
> > Client: \encoding returns 'UNICODE'.
> > Server: \list show databases. All databases are UNICODE (except TEMPLATE0
> > and TEMPLATE1 which are ASCII of course). I use a Mandrake 8.1 
> distribution
> > and think my console is UNICODE.
>
>You think?  Try this:
>
>         $ echo "accepté" | od -c
>
>If your term is in utf you should get:
>
>         0000000   a   c   c   e   p   t 303 251  \n
>         0000011
>
>If in iso-8859-1:
>
>         0000000   a   c   c   e   p   t 351  \n
>         0000010
>
>It may be in some other 8bit encoding too, then the last number
>may be different.
It is:
         0000000   a   c   c  e   p   t     é \n
         0000010


>Hmm.  It may be a bug in input routines.  You give PostgreSQL a
>1byte 'é', it expects 2 byte char and overflows somewhere.  Can
>you reproduce it on 7.1.3?  Maybe its fixed there, I cant
>reproduce it.

I noticed some longer routines with "é" worked without any problem.
I cannot reproduce it as I converted my database to plain ASCII.
Will try UNICODE on 7.2 beta when adding Japanese text to my database.

Thank you very much for your help.
Best regards, Jean-Michel POURE





^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 15:59  Marko Kreen <marko@l-t.ee>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  0 siblings, 0 replies; 23+ messages in thread

From: Marko Kreen @ 2001-10-28 15:59 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: pgsql-hackers@postgresql.org

On Sun, Oct 28, 2001 at 04:37:48PM +0100, Jean-Michel POURE wrote:
> At 17:09 28/10/01 +0200, you wrote:
> >        $ echo "accepté" | od -c
> It is:
>         0000000   a   c   c  e   p   t     é \n
>         0000010

Huh.  Then try 'od -t x1'.  Also what the commend 'locale'
prints.


> >Hmm.  It may be a bug in input routines.  You give PostgreSQL a
> >1byte 'é', it expects 2 byte char and overflows somewhere.  Can
> >you reproduce it on 7.1.3?  Maybe its fixed there, I cant
> >reproduce it.
> 
> I noticed some longer routines with "é" worked without any problem.
> I cannot reproduce it as I converted my database to plain ASCII.
> Will try UNICODE on 7.2 beta when adding Japanese text to my database.

Ok.  I still suggest you try to understand what was going on,
otherwise you will be in trouble again.  The logic around
encodings will be same in 7.2.

-- 
marko




^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-28 18:03  Patrice <phede-ml@islande.org>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  2 siblings, 1 reply; 23+ messages in thread

From: Patrice Hédé @ 2001-10-28 18:03 UTC (permalink / raw)
  To: Jean-Michel POURE <jm.poure@freesurf.fr>; +Cc: Marko Kreen <marko@l-t.ee>; pgsql-hackers@postgresql.org

Hi Jean-Micehl,

* Jean-Michel POURE <jm.poure@freesurf.fr> [011028 18:23]:
> 
> >psql uses your input literally - so is your console/xterm in
> >UNICODE/UTF8?
> Client: \encoding returns 'UNICODE'.
> Server: \list show databases. All databases are UNICODE (except
> TEMPLATE0 and TEMPLATE1 which are ASCII of course). I use a Mandrake
> 8.1 distribution and think my console is UNICODE.

I don't know the details for the Mandrake distribution, but I would
rather think the default terminal to be iso-8859-15 or iso-8859-1
encoded (I use myself a linux debian sid, customised to be mixed
iso-8859-15/utf-8 :) ).

In that case, it's likely to cause problems.
One thing is to check your current locale (before running psql), by
typing "locale charmap" on your terminal :

Unicode :

asterix:~$ locale charmap
UTF-8

latin-9 (fr_FR@euro) :

asterix:~$ locale charmap
ISO-8859-15

Then, if you really have a Unicode term, then you may run into other
problems. Psql uses readline, and readline is not yet "utf-8" enabled
by default. There are patches for that, but I don't know why they
don't integrate the support into the code... whatever the reason, it
means that for example Backspace won't work over characters with more
than one byte, and that includes everything which is not ASCII.

So, if while typing in psql, you try to do some text editing over the
"é", then it's likely to mangle your input to psql (without
necessarily be visible in your terminal), and anything from a bad
commandline, to psql waiting for more input... When you've finished
typing your line, check if psql prompt is displaying an "=" sign :

tests=#

Third, depending on how your data is entered vs queried, it may have
some differences. For example, if you use an application which
converts UTF-8 data to D-normalisation before submitting to
PostgreSQL, then the "é" will be stored as "e"+"combining mark acute
accent". Then, when you do your query, you have to submit in the same
format, as "é" (directly typed from the keyboard) and "e"+"comb.acute
accent" are two different things (I plan to add support in PostgreSQL
for this kind of stuff for 7.3, if I manage to go a bit faster on my
other projects...).

Anyway, I have been trying a query like yours, using a UTF-8 xterm,
with a UNICODE encoding, both psql and database :

my table :

tests=# insert into matable values ('un texte accentué', 12);
INSERT 70197 1
tests=# insert into matable values ('ça accentue le problème', 14);
INSERT 70198 1

tests=# select * from matable;
         montext         | valeur 
-------------------------+--------
 un texte accentué       |     12
 ça accentue le problème |     14
(2 rows)

[note that the "é", "ç" and "è" are not combining forms here...]

tests=# select * from matable where montext ilike '%accentué%';
      montext      | valeur 
-------------------+--------
 un texte accentué |     12
(1 row)

It works fine for me.

> >> As for me, I typed INSERT INTO source_content VALUES ('Permis de
> >> conduire accepté') in Psql.
> >As I said - psql does not do any conversion.
> The faulty query is: INSERT INTO test (source_content) VALUES
> ('Permis de conduire accepté');
> 
> I just can't believe that Psql is not UTF-8 compatible. It seems
> unreal as Psql is PostgreSQL #1 helper application. Should I use
> PostgreSQL MULE encoding to have automatic trans coding. What are
> the guidelines, I am completely lost.

Psql is UTF-8 compatible. However, the terminal support of UTF-8 may
be a little shaky for now (no dead keys, no compose key) and that will
be fixed in Xfree-4.2, and readline support of UTF-8 is deficient (as
is bash's, where readline comes from). I don't know when *that* will
be fixed. I know http://www.li18nux.org/ has some patches, but I
haven't tried them yet.

> >> Psql does not insert the data and I have to kill it manually. Can
> >> you reproduce this?
> >No.  If it hangs this is serious problem.  Or did you simply forgot
> >final ';' ?   It btw does not seem valid sql to me, considering you
> >previously provided table structure.
> Is it possible that my database is corrupted? I have used pg_dump
> several times to dump data from production server to development
> servers and conversely. Does pg_dump produce UTF8 output? What are
> the guidelines when using UTF-8: forget psql and pg_dump?

One thing you really have to be careful about is the locale you're
running your terminal into (cf above with "locale charmap"). A lot of
tools are sensitive to that, as soon as they set the locale, and also
the terminal itself is sensitive to that (if you run an xterm, a
gnome-terminal or other, make sure they are started themselves with
the correct locale, rather than the locale being set by a .bashrc or
.profile AFTER the xterm is launched. One way to be sure is to launch
an Xterm from the command line in an other xterm ;) ).

> >In the end: are the strings/queries you give to psql/pg_exec UTF-8
> >- this is now main thing, as you have _configured_ everything
> >correctly.
> Everything is configured correctly server-side (PostgreSQL, Psql).
> 
> Thank you very much for your support Marko,
> Best regards,
> Jean-Michel

It's possible to work with psql and UTF-8, I'm using it :) But support
for utf-8 is not complete yet, and it's not seamless. Also, support in
Postgresql is not yet complete for UTF-8 (normalisation forms,
collation, regexes...), but it'll come :)

Patrice.

-- 
Patrice Hédé
email: patrice hede à islande org
www  : http://www.islande.org/



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-29 00:57  Tatsuo Ishii <t-ishii@sra.co.jp>
  parent: Jean-Michel POURE <jm.poure@freesurf.fr>
  4 siblings, 0 replies; 23+ messages in thread

From: Tatsuo Ishii @ 2001-10-29 00:57 UTC (permalink / raw)
  To: jm.poure@freesurf.fr; +Cc: pgsql-hackers@postgresql.org

I'm not sure what you are expecting but...

> SELECT * FROM test WHERE source_content ILIKE '%accept%';
  ---> returns a record

> SELECT * FROM test WHERE source_content ILIKE '%accepté%'
  ---> returns a record

> SELECT * FROM test WHERE source_content ILIKE '%accepte%'
  ---> returns 0 record

So all of above seem to work fine for me.

$ pg_config --configure
--prefix=/usr/local/pgsql --enable-multibyte=EUC_JP --enable-unicode-conversion --with-tcl --with-perl --enable-syslog --enable-debug --with-CXX --with-java
$ pg_config --version
PostgreSQL 7.1.3
--
Tatsuo Ishii



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: UNICODE
@ 2001-10-29 00:58  Tatsuo Ishii <t-ishii@sra.co.jp>
  parent: Patrice <phede-ml@islande.org>
  0 siblings, 0 replies; 23+ messages in thread

From: Tatsuo Ishii @ 2001-10-29 00:58 UTC (permalink / raw)
  To: phede-ml@islande.org; +Cc: jm.poure@freesurf.fr; marko@l-t.ee; pgsql-hackers@postgresql.org

> In that case, it's likely to cause problems.
> One thing is to check your current locale (before running psql), by
> typing "locale charmap" on your terminal :
> 
> Unicode :
> 
> asterix:~$ locale charmap
> UTF-8

Just curious. Are there any working charmap for UTF-8? I mean, the
charmap contains not only ISO 8859-* but also other languages defined
in UNICODE 2.0 at least. I coudn't find such a thing around me. Also,
does it handle Unicode combined characters?
--
Tatsuo Ishii



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Unicode!
@ 2005-03-11 01:20  star star <hoatamet@yahoo.com>
  0 siblings, 2 replies; 23+ messages in thread

From: star star @ 2005-03-11 01:20 UTC (permalink / raw)
  To: pgsql-admin

I use PostgreSQL 8!

I have 2 file: 
 
insert.htm
<html>
<head>
<title>Untitled Document</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<form name="form1" method="post" action="postgre_insert.jsp" >
  <input type="text" name="username">
  <input type="submit" name="Submit" value="Submit">
</form>
</body>
</html>
 
postgre_insert.jsp
<%@ page  contentType="text/html; charset=utf-8" import="java.sql.*" errorPage="" %>
<% 
 String username=request.getParameter("username");
 String str="insert into user_table values('"+username+"','try')"; 
 Class.forName("org.postgresql.Driver"); 
    String url = "jdbc:postgresql://localhost:5432/try"; 
    Connection conn = DriverManager.getConnection(url, "postgres", "pass"); 
    PreparedStatement s = conn.prepareStatement(str);
 s.executeUpdate(); 
 out.println("OK");
%>
I input "Ti#7871;ng Vi#7879;t" on textfield and click Submit button. Insertion is successed.
But, when i view data in pgAdmin III(version 1.2.0), data is " Tiếng Viá»#135;t ". 
Can you help me!?

		
---------------------------------
Do you Yahoo!?
 Yahoo! Small Business - Try our new resources site! 

^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: Unicode!
@ 2005-03-11 07:14  Miroslav Šulc <miroslav.sulc@startnet.cz>
  parent: star star <hoatamet@yahoo.com>
  1 sibling, 0 replies; 23+ messages in thread

From: Miroslav Šulc @ 2005-03-11 07:14 UTC (permalink / raw)
  To: star star <hoatamet@yahoo.com>; +Cc: pgsql-admin

First, I would try to set encoding (charset) for the insert.htm page to 
UTF-8. That might help. If not, let us know.

Miroslav Šulc


star star wrote:

> I use PostgreSQL 8!
>
> I have 2 file:
> insert.htm
> <html>
> <head>
> <title>Untitled Document</title>
> <meta http-equiv="Content-Type" content="text/html; ">
> </head>
> <body>
> <form name="form1" method="post" action="postgre_insert.jsp" >
> <input type="text" name="username">
> <input type="submit" name="Submit" value="Submit">
> </form>
> </body>
> </html>
> postgre_insert.jsp
> <%@ page contentType="text/html; charset=utf-8" import="java.sql.*" 
> errorPage="" %>
> <%
> String username=request.getParameter("username");
> String str="insert into user_table values('"+username+"','try')";
> Class.forName("org.postgresql.Driver");
> String url = "jdbc:postgresql://localhost:5432/try";
> Connection conn = DriverManager.getConnection(url, "postgres", "pass");
> PreparedStatement s = conn.prepareStatement(str);
> s.executeUpdate();
> out.println("OK");
> %>
> I input "Tiếng Việt" on textfield and click Submit button. Insertion 
> is successed.
> But, when i view data in pgAdmin III(version 1.2.0), data is " Tiếng 
> Việt ".
> Can you help me!?
>
> ------------------------------------------------------------------------
> Do you Yahoo!?
> Yahoo! Small Business - Try our new resources site! 
> <http://us.rd.yahoo.com/evt=31637/*http://smallbusiness.yahoo.com/resources/; 

Attachments:

miroslav.sulc.vcf (text/x-vcard, 401B) download begin:vcard fn;quoted-printable:Miroslav =C5=A0ulc n;quoted-printable:=C5=A0ulc;Miroslav org:StartNet s.r.o. adr;quoted-printable;quoted-printable:;;Vrchlick=C3=A9ho 161/5;Praha 5;;150 00;=C4=8Cesk=C3=A1 republika email;internet:miroslav.sulc@startnet.cz title:CEO tel;work:+420 257 225 602 tel;cell:+420 603 711 413 x-mozilla-html:TRUE url:http://www.startnet.cz version:2.1 end:vcard
^ permalink raw reply [nested|flat] 23+ messages in thread

* Re: Unicode!
@ 2005-03-16 07:52  Chen Shaopeng <chen_shaopeng@idsignet.com>
  parent: star star <hoatamet@yahoo.com>
  1 sibling, 0 replies; 23+ messages in thread

From: Chen Shaopeng @ 2005-03-16 07:52 UTC (permalink / raw)
  To: star star <hoatamet@yahoo.com>; +Cc: pgsql-admin

Since you used Java JDBC to insert the data into PostgreSQL,
can you also JDBC to pull it out back and see if you get
the right data back? If not, then your database encoding
is probably the error. You need to set the DB encoding
properly.

If you can get your data back to the correct encoding,
then check the environment settings of your pgadmin III.
I don't quite remember which version of pgadmin always
gave me that problem too when I used it in chinese
environment. My data in the DB is correct, and I can
get it back in the right encoding, but pgadmin just does
not display correctly, even though other applications
running in the same environment settings can display
unicode correctly.

Not sure if you are running Linux or Windows. If you run
Linux, you can set your environments to UTF-8, then connect
to the DB with psql, and display the contents of your data.
If your data are in unicode, you should see the display
correctly.

csp

在 2005-03-10四的 17:20 -0800,star star写道:
> I use PostgreSQL 8!
> 
> I have 2 file: 
>  
> insert.htm
> <html>
> <head>
> <title>Untitled Document</title>
> <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
> </head>
> <body>
> <form name="form1" method="post" action="postgre_insert.jsp" >
>   <input type="text" name="username">
>   <input type="submit" name="Submit" value="Submit">
> </form>
> </body>
> </html>
>  
> postgre_insert.jsp
> <%@ page  contentType="text/html; charset=utf-8" import="java.sql.*"
> errorPage="" %>
> <% 
>  String username=request.getParameter("username");
>  String str="insert into user_table values('"+username+"','try')"; 
>  Class.forName("org.postgresql.Driver"); 
>     String url = "jdbc:postgresql://localhost:5432/try"; 
>     Connection conn = DriverManager.getConnection(url, "postgres",
> "pass"); 
>     PreparedStatement s = conn.prepareStatement(str);
>  s.executeUpdate(); 
>  out.println("OK");
> %>
> I input "Ti�6�5ng Vi�6�3t" on textfield and click Submit button. Insertion
> is successed.
> But, when i view data in pgAdmin III(version 1.2.0), data is " Ting
> Vi�6�0t ". 
> Can you help me!?
> 

-- 
陈少鹏 Chen Shaopeng
上海龙方信息技术有限公司
http://www.idsignet.com

Attachments:

  signature.asc (application/pgp-signature, 188B) download

^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* unicode
@ 2005-06-24 09:47  Hannes Dorbath <light@theendofthetunnel.de>
  0 siblings, 1 reply; 23+ messages in thread

From: Hannes Dorbath @ 2005-06-24 09:47 UTC (permalink / raw)
  To: pgsql-admin

Hi,
can someone summarize the steps necessary to get PG8 fully setup for 
UTF-8 use, please? Locale, initDB params? Especially that SQL functions 
like UPPER, LOWER, LENGTH etc work correct as well..

Thanks in Advance



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-24 14:55  Peter Eisentraut <peter_e@gmx.net>
  parent: Hannes Dorbath <light@theendofthetunnel.de>
  0 siblings, 1 reply; 23+ messages in thread

From: Peter Eisentraut @ 2005-06-24 14:55 UTC (permalink / raw)
  To: Hannes Dorbath <light@theendofthetunnel.de>; +Cc: pgsql-admin

Am Freitag, 24. Juni 2005 11:47 schrieb Hannes Dorbath:
> can someone summarize the steps necessary to get PG8 fully setup for
> UTF-8 use, please? Locale, initDB params? Especially that SQL functions
> like UPPER, LOWER, LENGTH etc work correct as well..

Use initdb --locale=de_DE.utf8 and that should be all.



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-30 09:58  Hannes Dorbath <light@theendofthetunnel.de>
  parent: Peter Eisentraut <peter_e@gmx.net>
  0 siblings, 1 reply; 23+ messages in thread

From: Hannes Dorbath @ 2005-06-30 09:58 UTC (permalink / raw)
  To: pgsql-admin

Thanks. This works fine on Linux, but I could't get it to work on 
FreeBSD 5.3 - any ideas?

On 24.06.2005 16:55, Peter Eisentraut wrote:
> 
> Use initdb --locale=de_DE.utf8 and that should be all.
> 



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-30 10:54  Peter Eisentraut <peter_e@gmx.net>
  parent: Hannes Dorbath <light@theendofthetunnel.de>
  0 siblings, 1 reply; 23+ messages in thread

From: Peter Eisentraut @ 2005-06-30 10:54 UTC (permalink / raw)
  To: Hannes Dorbath <light@theendofthetunnel.de>; +Cc: pgsql-admin

Am Donnerstag, 30. Juni 2005 11:58 schrieb Hannes Dorbath:
> Thanks. This works fine on Linux, but I could't get it to work on
> FreeBSD 5.3 - any ideas?

I hear that FreeBSD doesn't support Unicode, so you're probably out of luck 
for now.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-30 11:26  Hannes Dorbath <light@theendofthetunnel.de>
  parent: Peter Eisentraut <peter_e@gmx.net>
  0 siblings, 1 reply; 23+ messages in thread

From: Hannes Dorbath @ 2005-06-30 11:26 UTC (permalink / raw)
  To: pgsql-admin

> I hear that FreeBSD doesn't support Unicode, so you're probably out of luck 
> for now.

Hm, but the locales are there, it's just named de_DE.UTF-8 instead of 
de_DE.utf8 on FreeBSD. InitDB crashs with the following:

[pgsql@kirk pgsql]$initdb --locale=de_DE.UTF-8
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale de_DE.UTF-8.
The default database encoding has accordingly been set to UNICODE.

fixing permissions on existing directory /opt/pgsql/data ... ok
creating directory /opt/pgsql/data/global ... ok
creating directory /opt/pgsql/data/pg_xlog ... ok
creating directory /opt/pgsql/data/pg_xlog/archive_status ... ok
creating directory /opt/pgsql/data/pg_clog ... ok
creating directory /opt/pgsql/data/pg_subtrans ... ok
creating directory /opt/pgsql/data/base ... ok
creating directory /opt/pgsql/data/base/1 ... ok
creating directory /opt/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /opt/pgsql/data/base/1 ... FATAL:  XX000: 
failed to initialize lc_messages to ""
LOCATION:  InitializeGUCOptions, guc.c:2389
child process exited with exit code 1
initdb: removing contents of data directory "/opt/pgsql/data"
[pgsql@kirk pgsql]$

-- 
imos  Gesellschaft fuer Internet-Marketing und Online-Services mbH
Alfons-Feifel-Str. 9 // D-73037 Goeppingen  // Stauferpark Ost
Tel: 07161 93339-14 // Fax: 07161 93339-99 // Internet: www.imos.net



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-30 13:57  Tom Lane <tgl@sss.pgh.pa.us>
  parent: Hannes Dorbath <light@theendofthetunnel.de>
  0 siblings, 1 reply; 23+ messages in thread

From: Tom Lane @ 2005-06-30 13:57 UTC (permalink / raw)
  To: Hannes Dorbath <light@theendofthetunnel.de>; +Cc: pgsql-admin

Hannes Dorbath <light@theendofthetunnel.de> writes:
>> I hear that FreeBSD doesn't support Unicode, so you're probably out of luck 
>> for now.

> Hm, but the locales are there, it's just named de_DE.UTF-8 instead of 
> de_DE.utf8 on FreeBSD. InitDB crashs with the following:

> creating template1 database in /opt/pgsql/data/base/1 ... FATAL:  XX000: 
> failed to initialize lc_messages to ""
> LOCATION:  InitializeGUCOptions, guc.c:2389

We've seen that happen before.  As far as I can see it indicates
breakage in the locale library: setlocale(LC_MESSAGES, "")
should work, but it's failing.

I'm not certain this is the total story, since the same failure
has been seen on several platforms and it's hard to believe they
are all brain-damaged.  But I don't really see what we should
do differently.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 23+ messages in thread

* Re: unicode
@ 2005-06-30 14:28  Hannes Dorbath <light@theendofthetunnel.de>
  parent: Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 23+ messages in thread

From: Hannes Dorbath @ 2005-06-30 14:28 UTC (permalink / raw)
  To: pgsql-admin

After 7 hours of trail an error I upgraded FreeBSD from 5.3 to 5.4 - and 
all of sudden - everything works :)

On 30.06.2005 15:57, Tom Lane wrote:
> We've seen that happen before.  As far as I can see it indicates
> breakage in the locale library: setlocale(LC_MESSAGES, "")
> should work, but it's failing.
> 
> I'm not certain this is the total story, since the same failure
> has been seen on several platforms and it's hard to believe they
> are all brain-damaged.  But I don't really see what we should
> do differently.



^ permalink  raw  reply  [nested|flat] 23+ messages in thread


end of thread, other threads:[~2005-06-30 14:28 UTC | newest]

Thread overview: 23+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2001-10-28 08:22 UNICODE Jean-Michel POURE <jm.poure@freesurf.fr>
2001-10-28 09:50 ` Marko Kreen <marko@l-t.ee>
2001-10-28 10:06 ` Jean-Michel POURE <jm.poure@freesurf.fr>
2001-10-28 10:53 ` Marko Kreen <marko@l-t.ee>
2001-10-28 11:44 ` Jean-Michel POURE <jm.poure@freesurf.fr>
2001-10-28 12:05   ` Marko Kreen <marko@l-t.ee>
2001-10-28 13:34   ` Jean-Michel POURE <jm.poure@freesurf.fr>
2001-10-28 15:09     ` Marko Kreen <marko@l-t.ee>
2001-10-28 15:37     ` Jean-Michel POURE <jm.poure@freesurf.fr>
2001-10-28 15:59       ` Marko Kreen <marko@l-t.ee>
2001-10-28 18:03     ` Patrice <phede-ml@islande.org>
2001-10-29 00:58       ` Tatsuo Ishii <t-ishii@sra.co.jp>
2001-10-29 00:57 ` Tatsuo Ishii <t-ishii@sra.co.jp>
2005-03-11 01:20 Unicode! star star <hoatamet@yahoo.com>
2005-03-11 07:14 ` Re: Unicode! Miroslav Šulc <miroslav.sulc@startnet.cz>
2005-03-16 07:52 ` Re: Unicode! Chen Shaopeng <chen_shaopeng@idsignet.com>
2005-06-24 09:47 unicode Hannes Dorbath <light@theendofthetunnel.de>
2005-06-24 14:55 ` Re: unicode Peter Eisentraut <peter_e@gmx.net>
2005-06-30 09:58   ` Re: unicode Hannes Dorbath <light@theendofthetunnel.de>
2005-06-30 10:54     ` Re: unicode Peter Eisentraut <peter_e@gmx.net>
2005-06-30 11:26       ` Re: unicode Hannes Dorbath <light@theendofthetunnel.de>
2005-06-30 13:57         ` Re: unicode Tom Lane <tgl@sss.pgh.pa.us>
2005-06-30 14:28           ` Re: unicode Hannes Dorbath <light@theendofthetunnel.de>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox