public inbox for [email protected]
help / color / mirror / Atom feed[Pljava-dev] PL/java kills unicode chars?
4+ messages / 1 participants
[nested] [flat]
* [Pljava-dev] PL/java kills unicode chars?
@ 2015-09-20 03:41
2015-09-23 23:39 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
0 siblings, 1 reply; 4+ messages in thread
From: @ 2015-09-20 03:41 UTC (permalink / raw)
Srivatsan Ramanujam wrote:
> I believe PL/java is killing unicode characters (it is probably converting
> text to a byte stream and reading them as single byte characters - perhaps
> Latin-1 and not as UTF-8). ...
Well, this one has been open for a while. It's also in the github tracker,
https://github.com/tada/pljava/issues/21 which I've just updated with
confirming test code.
I think vatsan's comment (on the github issue) about
http://bugs.sun.com/view_bug.do?bug_id=5030776 is probably spot on.
We seem to handle the whole basic multilingual plane correctly
(nearly 64k codepoints), it's just all the planes above that getting
messed up.
Should be a straightforward fix, once I've found how many places those
not-really-UTF JNI functions are really used in the code.
-Chap
^ permalink raw reply [nested|flat] 4+ messages in thread
* [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-20 03:41 [Pljava-dev] PL/java kills unicode chars?
@ 2015-09-23 23:39 `
2015-09-24 06:21 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
0 siblings, 1 reply; 4+ messages in thread
From: @ 2015-09-23 23:39 UTC (permalink / raw)
Question (mostly for Thomas),
In working on issue 21, would it be acceptable to add specialized
calls (in JNICalls or elsewhere), for calling up into a very select
few Java methods without releasing and reacquiring the threadlock?
such as...
CharSequence.toString()
java.nio.Buffer.position()
java.nio.CharBuffer.wrap(CharSequence)
java.nio.charset.CharsetDecoder.decode(ByteBuffer)
java.nio.charset.CharsetEncoder.encode(CharBuffer,ByteBuffer,boolean)
They could be called rather often ;) and the monitor operations could be
overkill; no point giving another thread access to the backend during
such a simple operation.
I like your comments in the source saying "Would be nice if a direct
conversion from UTF16 was provided." This task is like some kind of
"you can't get there from here" Vaudeville routine. Between what's
provided in PG and what's in Java there are at least three different APIs
to approach the problem with, and for *each one* there seems to be *one*
feature lacking that necessitates going the long way around, too many
pallocs/copies. etc.
Or Kafka. It could be Kafka.
-Chap
^ permalink raw reply [nested|flat] 4+ messages in thread
* [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-20 03:41 [Pljava-dev] PL/java kills unicode chars?
2015-09-23 23:39 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
@ 2015-09-24 06:21 `
2015-09-24 23:34 ` Re: [Pljava-dev] Issue 21 Re: PL/java kills unicode chars? Chapman Flack <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: @ 2015-09-24 06:21 UTC (permalink / raw)
Hi Chap,
I'm not sure what you're asking here. It is always OK to use methods in the Java library without a thread lock provided
there's absolutely no chance that any new threads that they might start calls back into the backend. Don't see how any
of the methods you mention could do that.
Or perhaps you mean that you would like to call these methods from SQL without imposing a thread lock? In that case,
yes, that's OK too as long as they in turn do not call anything that a user has written and obey by the rule stated above.
That said, I'm very much in favor of solutions that doesn't require the user to have specific knowledge about encodings.
Once configured, it should just work regardless. I.e. conversion to/from UTF should ideally be invisible and no special
calls from SQL should be needed.
In order to advice on a good solution I'd like to first know exactly what the problem and it's possible solutions are.
- thomas
On 2015-09-24 01:39, Chapman Flack wrote:
> Question (mostly for Thomas),
>
> In working on issue 21, would it be acceptable to add specialized
> calls (in JNICalls or elsewhere), for calling up into a very select
> few Java methods without releasing and reacquiring the threadlock?
>
> such as...
>
> CharSequence.toString()
> java.nio.Buffer.position()
> java.nio.CharBuffer.wrap(CharSequence)
> java.nio.charset.CharsetDecoder.decode(ByteBuffer)
> java.nio.charset.CharsetEncoder.encode(CharBuffer,ByteBuffer,boolean)
>
> They could be called rather often ;) and the monitor operations could be
> overkill; no point giving another thread access to the backend during
> such a simple operation.
>
> I like your comments in the source saying "Would be nice if a direct
> conversion from UTF16 was provided." This task is like some kind of
> "you can't get there from here" Vaudeville routine. Between what's
> provided in PG and what's in Java there are at least three different APIs
> to approach the problem with, and for *each one* there seems to be *one*
> feature lacking that necessitates going the long way around, too many
> pallocs/copies. etc.
>
> Or Kafka. It could be Kafka.
>
> -Chap
> _______________________________________________
> Pljava-dev mailing list
> Pljava-dev at lists.pgfoundry.org
> http://lists.pgfoundry.org/mailman/listinfo/pljava-dev
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-20 03:41 [Pljava-dev] PL/java kills unicode chars?
2015-09-23 23:39 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-24 06:21 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
@ 2015-09-24 23:34 ` Chapman Flack <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Chapman Flack @ 2015-09-24 23:34 UTC (permalink / raw)
Thomas Hallgren wrote:
> I'm not sure what you're asking here. It is always OK to use methods in
> the Java library without a thread lock provided there's absolutely no
> chance that any new threads that they might start calls back into the
> backend. Don't see how any of the methods you mention could do that.
Great, so far so good, that was a big part of my question. (Not purely
whether it was _technically_ ok, which is a question I could answer,
but also whether it would be satisfactory to you under the coding
conventions you have for PL/Java.)
Right now, all calls into the JNI seem to go through the wrapper methods
in JNICalls.c, which at the very least all play the BEGIN_JAVA/END_JAVA
shell-game with the JNIEnv pointer ... and if they are method invocations,
they further do the exit and reentry of the monitor. So if I wanted
to do more lightweight invocations only for charset encode/decode methods,
it looks as if somewhere (maybe in JNICalls.c) I would need to add
alternate method-invocation wrappers that do most of the same work but
leave out the monitor operations. I didn't know whether you would
consider that undesirable.
> In order to advice on a good solution I'd like to first know exactly
> what the problem and it's possible solutions are.
Well, so at bottom the problem is PL/Java treating the JNI methods
newStringUTF and getStringUTFChars as if they really meant UTF-8.
Of course they are really the JNI "modified UTF-8" which is much
closer to CESU-8 (except for also having the NUL hack, so it's not
quite that either).
By contrast, when you ask PostgreSQL for UTF-8, as with
pg_do_encoding_conversion( ... PG_UTF8 ... ), you get real, true,
no-funny-business UTF-8. The trouble is, that's not quite what should
be exchanged with JNI.
Nobody ever used to notice, back in the day when there weren't any
characters anyone cared about outside the first 64k Unicode plane.
But these days there are, as the person reporting issue 21 found out,
and that's where the problem shows up. As the test code I added to
issue 21 shows, all 1024 other 64k character code blocks above block 0
corrupted in a round trip from PL/Java through Java ... because UTF-8
is not CESU-8.
Possible solutions fall in some basic classes:
I. Use real UTF-8 consistently. Call the pg_ routine to get utf-8, as
currently done in String.c, but don't use the JNI newStringUTF/
getStringUTFChars (which are really CESU-8); instead use the
java.lang.String bytes constructor and get bytes methods with explicit
UTF-8 charset, because that really *is* UTF-8. Lower-level techniques
like using the java.nio.charset UTF-8 converters directly, to try to
reduce allocations and copies, I would also consider in Class I.
II. Use real CESU-8 consistently. Write a CESU-8 codec that can fit
into pg's converter scheme so you could just ask for that encoding from
pg and pass it straight to the JNI routines. Nice idea, but brick walls:
PG has a *hard-coded* set of encodings it knows. Funnily enough, it has
an API that lets you define new converters *between* encodings ... as long
as you are only talking about the encodings it already knows. Also, it
seems to have hardcoded assumptions that no encoding can make more than
4 bytes from a codepoint, where CESU-8 can sometimes make 6. So class II
seems to be impractical.
III. Use real UTF-16 consistently. That's the other encoding that has direct
JNI support in and out of strings, and it might likely be the JVM's internal
representation, so if there were a pg_ conversion to get from the server
encoding to UTF-16, the JNI calls in and out of strings should Just Work,
and quite possibly would work without incurring an extra conversion. Nice
work if you can get it, but there isn't a UTF-16 codec in pg, so it ends
up the same place as class II.
IV. Use real UCS-4 consistently. That's the pg_wchar encoding and there
are functions and macros provided by pg for converting between UTF-8 and
wchar, either character-at-a-time or whole-string-at-a-time. In recent
Javas, String does have a constructor from int[], and a method
getCodePointAt you can call in a loop, but no int[] getCodePoints().
In pg there is even a table giving foo <-> wchar converters for other
server encodings, so it could be possible to skip the intermediate UTF-8
step and go straight from server encoding to UCS-4 and build the string
from that. What's the problem with that? The only coders you find in the
table are the whole-string-at-a-time ones, and they *don't have any
parameter for output buffer length*! You have to guess, worst-case
expansion, and preallocate the whole thing. (Or allocate at the end of
memory, and catch SIGSEGV....) And the worst-case expansion is a lot
bigger than the average case.
I've shifted my affections over the last week among all four of those
classes, but I am currently leaning toward class I. (From some googling,
that seems to be a common solution for other Java devs too.) I *am*
interested in using the lower-level methods in the hope of at least
slightly reducing the reallocating and copying involved.
Class I could also become a class I' where you find out if the JVM knows
a coding that corresponds directly to the server encoding, and skip the
UTF-8 step. I am not planning to implement that, at least not at first.
In the case of server encoding == utf-8, that essentially happens anyway,
since the pg_do_encoding_conversion... makes itself a no-op in that case.
I assume utf-8 is the usual server encoding these days, so for a first
effort that sounds "good enough". It will still be correct for other
encodings, just slower.
-Chap
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2015-09-24 23:34 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2015-09-20 03:41 [Pljava-dev] PL/java kills unicode chars?
2015-09-23 23:39 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-24 06:21 ` [Pljava-dev] Issue 21 Re: PL/java kills unicode chars?
2015-09-24 23:34 ` Re: [Pljava-dev] Issue 21 Re: PL/java kills unicode chars? Chapman Flack <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox