pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedJDBC encoding problem
5+ messages / 3 participants
[nested] [flat]
* JDBC encoding problem
@ 2003-08-05 17:11 Kurt Overberg <kurt@hotdogrecords.com>
1 sibling, 1 reply; 5+ messages in thread
From: Kurt Overberg @ 2003-08-05 17:11 UTC (permalink / raw)
To: pgsql-sql; pgsql-jdbc@postgresql.org
I'm having a rather strange problem that I'm hoping someone can help me
with. I'm using Struts 1.0/jsp on Debian linux under Tomcat 4.1.x and
the blackdown JVM . I'm attempting to convert my current SQL_ASCII
database to UNICODE. I'm new to this, so am most likely making a few
mistakes. Here's what I've done so far:
o Converted database encoding to be UNICODE. I'm pretty sure this part
worked okay. (did a pg_dump, then iconv -f 8859_1 -t UTF-8, then
created new db with encoding UNICODE and reloaded- no errors upon reload)
sparky:~$ psql -l
List of databases
Name | Owner | Encoding
-----------+----------+-----------
unitest | kurt | UNICODE
template1 | postgres | SQL_ASCII
(2 rows)
o set client_encoding to 'UTF8';
o In my JSP files, I set the following at the top of each:
<%@ page lanuage="java" pageEncoding="UTF-8" %>
Now, to test this, I go to a japanese page, copy some text, then paste
it into a form, that gets submitted to the server and saved into the DB.
Then I try to display what I got back from the database. It comes out
garbled. HOWEVER- if I leave the 'pageEncoding' out of my display .jsp
file it still comes out garbled, UNTIL I set UTF-8 manually in my
browsers Character Encoding settings (both mozilla and IE). Then the
japanese characters render fine (just like I entered them).
Very strange. What's confusing is that when I set the pageEncoding to
'UTF-8', the characters don't render properly, and as far as I can tell,
thats the same as manually setting the browser manually. I must be
doing something wrong because I get the same results in IE and mozilla
(recent build).
What may be the problem- I don't do anything differently when getting
the data out of the database, just standard
resultset.getString("column"); Do I need to change that call, to handle
the potentially UTF-8 encoded strings? I can't find anything on that at
all with google/usenet.
Any and all help, suggestions or pointers would be greatly appreciated.
Thanks!
/kurt
^ permalink raw reply [nested|flat] 5+ messages in thread
* JDBC encoding problem
@ 2003-08-05 17:31 Kurt Overberg <kurt@hotdogrecords.com>
1 sibling, 1 reply; 5+ messages in thread
From: Kurt Overberg @ 2003-08-05 17:31 UTC (permalink / raw)
To: pgsql-sql; pgsql-jdbc@postgresql.org
I'm having a rather strange problem that I'm hoping someone can help me
with. I'm using Struts 1.0/jsp on Debian linux under Tomcat 4.1.x and
the blackdown JVM . I'm attempting to convert my current SQL_ASCII
database to UNICODE. I'm new to this, so am most likely making a few
mistakes. Here's what I've done so far:
o Converted database encoding to be UNICODE. I'm pretty sure this part
worked okay. (did a pg_dump, then iconv -f 8859_1 -t UTF-8, then
created new db with encoding UNICODE and reloaded- no errors upon reload)
sparky:~$ psql -l
List of databases
Name | Owner | Encoding
-----------+----------+-----------
unitest | kurt | UNICODE
template1 | postgres | SQL_ASCII
(2 rows)
o set client_encoding to 'UTF8';
o In my JSP files, I set the following at the top of each:
<%@ page lanuage="java" pageEncoding="UTF-8" %>
Now, to test this, I go to a japanese page, copy some text, then paste
it into a form, that gets submitted to the server and saved into the DB.
Then I try to display what I got back from the database. It comes out
garbled. HOWEVER- if I leave the 'pageEncoding' out of my display .jsp
file it still comes out garbled, UNTIL I set UTF-8 manually in my
browsers Character Encoding settings (both mozilla and IE). Then the
japanese characters render fine (just like I entered them).
Very strange. What's confusing is that when I set the pageEncoding to
'UTF-8', the characters don't render properly, and as far as I can tell,
thats the same as manually setting the browser manually. I must be
doing something wrong because I get the same results in IE and mozilla
(recent build).
What may be the problem- I don't do anything differently when getting
the data out of the database, just standard
resultset.getString("column"); Do I need to change that call, to handle
the potentially UTF-8 encoded strings? I can't find anything on that at
all with google/usenet.
Any and all help, suggestions or pointers would be greatly appreciated.
Thanks!
/kurt
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: JDBC encoding problem
@ 2003-08-05 18:00 Anders Hermansen <anders@yoyo.no>
parent: Kurt Overberg <kurt@hotdogrecords.com>
0 siblings, 0 replies; 5+ messages in thread
From: Anders Hermansen @ 2003-08-05 18:00 UTC (permalink / raw)
To: Kurt Overberg <kurt@hotdogrecords.com>; +Cc: pgsql-jdbc@postgresql.org
* Kurt Overberg (kurt@hotdogrecords.com) wrote:
> I'm having a rather strange problem that I'm hoping someone can help me
> with. I'm using Struts 1.0/jsp on Debian linux under Tomcat 4.1.x and
> the blackdown JVM . I'm attempting to convert my current SQL_ASCII
> database to UNICODE. I'm new to this, so am most likely making a few
> mistakes. Here's what I've done so far:
>
> o Converted database encoding to be UNICODE. I'm pretty sure this part
> worked okay. (did a pg_dump, then iconv -f 8859_1 -t UTF-8, then
> created new db with encoding UNICODE and reloaded- no errors upon reload)
>
> sparky:~$ psql -l
> List of databases
> Name | Owner | Encoding
> -----------+----------+-----------
> unitest | kurt | UNICODE
> template1 | postgres | SQL_ASCII
> (2 rows)
Ok.
> o set client_encoding to 'UTF8';
As I read in another thread, client_encoding does not matter for the
JDBC driver. It will change it to UNICODE when you connect. It probably
does this because all java strings are unicode.
But it will probably matter for your psql connections, if any.
> o In my JSP files, I set the following at the top of each:
>
> <%@ page lanuage="java" pageEncoding="UTF-8" %>
Try to change this to
<%@ page lanuage="java" pageEncoding="UTF-8" contentType="text/html;charset=UTF-8" %>
> Now, to test this, I go to a japanese page, copy some text, then paste
> it into a form, that gets submitted to the server and saved into the DB.
> Then I try to display what I got back from the database. It comes out
> garbled. HOWEVER- if I leave the 'pageEncoding' out of my display .jsp
> file it still comes out garbled, UNTIL I set UTF-8 manually in my
> browsers Character Encoding settings (both mozilla and IE). Then the
> japanese characters render fine (just like I entered them).
>
> Very strange. What's confusing is that when I set the pageEncoding to
> 'UTF-8', the characters don't render properly, and as far as I can tell,
> thats the same as manually setting the browser manually. I must be
> doing something wrong because I get the same results in IE and mozilla
> (recent build).
>
> What may be the problem- I don't do anything differently when getting
> the data out of the database, just standard
> resultset.getString("column"); Do I need to change that call, to handle
> the potentially UTF-8 encoded strings? I can't find anything on that at
> all with google/usenet.
Have you tried putting unicode characters inside the db using pgsql? And
then showing them using the web-tier?
I have used tomcat as the webapp-server for many applications, and it
defaults to ISO-8859-1 character set for POST forms. Strange it is.
You can change this by calling
request.setCharacterEncoding("UTF-8");
before you get any data from your form.
Maybe the pageEncoding="UTF-8" changes this? I have not used that option
before.
Please check again that the data that you put in the database using JDBC
is not garbage due to characterset conversion.
> Any and all help, suggestions or pointers would be greatly appreciated.
I hope this helps,
Anders
--
Anders Hermansen
YoYo Mobile as
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: [SQL] JDBC encoding problem
@ 2003-08-06 01:10 LEON <ldai@accunettech.com>
parent: Kurt Overberg <kurt@hotdogrecords.com>
0 siblings, 1 reply; 5+ messages in thread
From: LEON @ 2003-08-06 01:10 UTC (permalink / raw)
To: Kurt Overberg <kurt@hotdogrecords.com>; pgsql-sql; pgsql-jdbc@postgresql.org
I have met the same problem when I deal with chinese.
I guess tomcat and jdbc produced this bug.
But you may solute the charset display problem.
You change the database's charset is useless when you connected DB with JDBC. You should convert Input Text with UTF-8 before saved in database. When you fetch data from DATABASE ,you should convert output data with some charset (F.E:GBK) before you display the data in page.
There is some java code I used.Hope they will help you.
//Output:
OutputStream fileOut = new FileOutputStream(outputFileName);
fileOut.write(data.getBytes("GBK")); //GBK、GB2312、BIG5、UTF8
//Input:
InputStream fileIn = new FileInputStream(inputFileName);
int i = fileIn.read(buff);
String data = new String(buff,"UTF8");
Best regards.
leon
----- Original Message -----
From: "Kurt Overberg" <kurt@hotdogrecords.com>
To: <pgsql-sql@postgresql.org>; <pgsql-jdbc@postgresql.org>
Sent: Wednesday, August 06, 2003 1:11 AM
Subject: [SQL] JDBC encoding problem
>
> I'm having a rather strange problem that I'm hoping someone can help me
> with. I'm using Struts 1.0/jsp on Debian linux under Tomcat 4.1.x and
> the blackdown JVM . I'm attempting to convert my current SQL_ASCII
> database to UNICODE. I'm new to this, so am most likely making a few
> mistakes. Here's what I've done so far:
>
> o Converted database encoding to be UNICODE. I'm pretty sure this part
> worked okay. (did a pg_dump, then iconv -f 8859_1 -t UTF-8, then
> created new db with encoding UNICODE and reloaded- no errors upon reload)
>
> sparky:~$ psql -l
> List of databases
> Name | Owner | Encoding
> -----------+----------+-----------
> unitest | kurt | UNICODE
> template1 | postgres | SQL_ASCII
> (2 rows)
>
>
> o set client_encoding to 'UTF8';
>
> o In my JSP files, I set the following at the top of each:
>
> <%@ page lanuage="java" pageEncoding="UTF-8" %>
>
>
> Now, to test this, I go to a japanese page, copy some text, then paste
> it into a form, that gets submitted to the server and saved into the DB.
> Then I try to display what I got back from the database. It comes out
> garbled. HOWEVER- if I leave the 'pageEncoding' out of my display .jsp
> file it still comes out garbled, UNTIL I set UTF-8 manually in my
> browsers Character Encoding settings (both mozilla and IE). Then the
> japanese characters render fine (just like I entered them).
>
> Very strange. What's confusing is that when I set the pageEncoding to
> 'UTF-8', the characters don't render properly, and as far as I can tell,
> thats the same as manually setting the browser manually. I must be
> doing something wrong because I get the same results in IE and mozilla
> (recent build).
>
> What may be the problem- I don't do anything differently when getting
> the data out of the database, just standard
> resultset.getString("column"); Do I need to change that call, to handle
> the potentially UTF-8 encoded strings? I can't find anything on that at
> all with google/usenet.
>
> Any and all help, suggestions or pointers would be greatly appreciated.
>
> Thanks!
>
> /kurt
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: [SQL] JDBC encoding problem
@ 2003-08-06 14:18 Kurt Overberg <kurt@hotdogrecords.com>
parent: LEON <ldai@accunettech.com>
0 siblings, 0 replies; 5+ messages in thread
From: Kurt Overberg @ 2003-08-06 14:18 UTC (permalink / raw)
To: LEON <ldai@accunettech.com>; +Cc: pgsql-sql; pgsql-jdbc@postgresql.org
It seems like I was able to get it to work with UTF-8. I found this
nifty little page: http://www.anassina.com/struts/i18n/i18n.html which
explained how to handle this within Jakarta Struts. It SEEMS like when
reading the data from the database I don't need to explictly convert it
to UTF-8, it just happens. When I converted my database, I dumped it
out, ran an ASCII to UNICODE converter, then loaded the data back in. I
was able to enter japanese characters and read them back from the DB
okay. My big question is completeness. I really have no idea if I'm
handling every case since it "just seems to work". Thanks for the reply!
/kurt
LEON wrote:
> I have met the same problem when I deal with chinese.
> I guess tomcat and jdbc produced this bug.
> But you may solute the charset display problem.
> You change the database's charset is useless when you connected DB with JDBC. You should convert Input Text with UTF-8 before saved in database. When you fetch data from DATABASE ,you should convert output data with some charset (F.E:GBK) before you display the data in page.
>
>
> There is some java code I used.Hope they will help you.
> //Output:
> OutputStream fileOut = new FileOutputStream(outputFileName);
>
> fileOut.write(data.getBytes("GBK")); //GBK、GB2312、BIG5、UTF8
>
> //Input:
> InputStream fileIn = new FileInputStream(inputFileName);
>
> int i = fileIn.read(buff);
>
> String data = new String(buff,"UTF8");
>
>
> Best regards.
> leon
>
> ----- Original Message -----
> From: "Kurt Overberg" <kurt@hotdogrecords.com>
> To: <pgsql-sql@postgresql.org>; <pgsql-jdbc@postgresql.org>
> Sent: Wednesday, August 06, 2003 1:11 AM
> Subject: [SQL] JDBC encoding problem
>
>
>
>>I'm having a rather strange problem that I'm hoping someone can help me
>>with. I'm using Struts 1.0/jsp on Debian linux under Tomcat 4.1.x and
>>the blackdown JVM . I'm attempting to convert my current SQL_ASCII
>>database to UNICODE. I'm new to this, so am most likely making a few
>>mistakes. Here's what I've done so far:
>>
>>o Converted database encoding to be UNICODE. I'm pretty sure this part
>>worked okay. (did a pg_dump, then iconv -f 8859_1 -t UTF-8, then
>>created new db with encoding UNICODE and reloaded- no errors upon reload)
>>
>>sparky:~$ psql -l
>> List of databases
>> Name | Owner | Encoding
>>-----------+----------+-----------
>> unitest | kurt | UNICODE
>> template1 | postgres | SQL_ASCII
>>(2 rows)
>>
>>
>>o set client_encoding to 'UTF8';
>>
>>o In my JSP files, I set the following at the top of each:
>>
>><%@ page lanuage="java" pageEncoding="UTF-8" %>
>>
>>
>>Now, to test this, I go to a japanese page, copy some text, then paste
>>it into a form, that gets submitted to the server and saved into the DB.
>>Then I try to display what I got back from the database. It comes out
>>garbled. HOWEVER- if I leave the 'pageEncoding' out of my display .jsp
>>file it still comes out garbled, UNTIL I set UTF-8 manually in my
>>browsers Character Encoding settings (both mozilla and IE). Then the
>>japanese characters render fine (just like I entered them).
>>
>>Very strange. What's confusing is that when I set the pageEncoding to
>>'UTF-8', the characters don't render properly, and as far as I can tell,
>>thats the same as manually setting the browser manually. I must be
>>doing something wrong because I get the same results in IE and mozilla
>>(recent build).
>>
>>What may be the problem- I don't do anything differently when getting
>>the data out of the database, just standard
>>resultset.getString("column"); Do I need to change that call, to handle
>>the potentially UTF-8 encoded strings? I can't find anything on that at
>>all with google/usenet.
>>
>>Any and all help, suggestions or pointers would be greatly appreciated.
>>
>>Thanks!
>>
>>/kurt
>>
>>
>>
>>---------------------------(end of broadcast)---------------------------
>>TIP 4: Don't 'kill -9' the postmaster
>>
>
>
> ---------------------------(end of broadcast)---------------------------TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@postgresql.org so that your message can get through to the mailing list cleanly
>
>
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2003-08-06 14:18 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2003-08-05 17:11 ` Kurt Overberg <kurt@hotdogrecords.com>
2003-08-06 01:10 ` LEON <ldai@accunettech.com>
2003-08-06 14:18 ` Kurt Overberg <kurt@hotdogrecords.com>
2003-08-05 17:31 ` Kurt Overberg <kurt@hotdogrecords.com>
2003-08-05 18:00 ` Anders Hermansen <anders@yoyo.no>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox