Message-ID: <5604886E.40104@anastigmatix.net> Date: Thu, 24 Sep 2015 19:34:06 -0400 From: Chapman Flack MIME-Version: 1.0 Subject: Re: [Pljava-dev] Issue 21 Re: PL/java kills unicode chars? References: <56039650.5090609@tada.se> In-Reply-To: <56039650.5090609@tada.se> Content-Type: text/plain; charset=ISO-8859-1 Thomas Hallgren wrote: > I'm not sure what you're asking here. It is always OK to use methods in > the Java library without a thread lock provided there's absolutely no > chance that any new threads that they might start calls back into the > backend. Don't see how any of the methods you mention could do that. Great, so far so good, that was a big part of my question. (Not purely whether it was _technically_ ok, which is a question I could answer, but also whether it would be satisfactory to you under the coding conventions you have for PL/Java.) Right now, all calls into the JNI seem to go through the wrapper methods in JNICalls.c, which at the very least all play the BEGIN_JAVA/END_JAVA shell-game with the JNIEnv pointer ... and if they are method invocations, they further do the exit and reentry of the monitor. So if I wanted to do more lightweight invocations only for charset encode/decode methods, it looks as if somewhere (maybe in JNICalls.c) I would need to add alternate method-invocation wrappers that do most of the same work but leave out the monitor operations. I didn't know whether you would consider that undesirable. > In order to advice on a good solution I'd like to first know exactly > what the problem and it's possible solutions are. Well, so at bottom the problem is PL/Java treating the JNI methods newStringUTF and getStringUTFChars as if they really meant UTF-8. Of course they are really the JNI "modified UTF-8" which is much closer to CESU-8 (except for also having the NUL hack, so it's not quite that either). By contrast, when you ask PostgreSQL for UTF-8, as with pg_do_encoding_conversion( ... PG_UTF8 ... ), you get real, true, no-funny-business UTF-8. The trouble is, that's not quite what should be exchanged with JNI. Nobody ever used to notice, back in the day when there weren't any characters anyone cared about outside the first 64k Unicode plane. But these days there are, as the person reporting issue 21 found out, and that's where the problem shows up. As the test code I added to issue 21 shows, all 1024 other 64k character code blocks above block 0 corrupted in a round trip from PL/Java through Java ... because UTF-8 is not CESU-8. Possible solutions fall in some basic classes: I. Use real UTF-8 consistently. Call the pg_ routine to get utf-8, as currently done in String.c, but don't use the JNI newStringUTF/ getStringUTFChars (which are really CESU-8); instead use the java.lang.String bytes constructor and get bytes methods with explicit UTF-8 charset, because that really *is* UTF-8. Lower-level techniques like using the java.nio.charset UTF-8 converters directly, to try to reduce allocations and copies, I would also consider in Class I. II. Use real CESU-8 consistently. Write a CESU-8 codec that can fit into pg's converter scheme so you could just ask for that encoding from pg and pass it straight to the JNI routines. Nice idea, but brick walls: PG has a *hard-coded* set of encodings it knows. Funnily enough, it has an API that lets you define new converters *between* encodings ... as long as you are only talking about the encodings it already knows. Also, it seems to have hardcoded assumptions that no encoding can make more than 4 bytes from a codepoint, where CESU-8 can sometimes make 6. So class II seems to be impractical. III. Use real UTF-16 consistently. That's the other encoding that has direct JNI support in and out of strings, and it might likely be the JVM's internal representation, so if there were a pg_ conversion to get from the server encoding to UTF-16, the JNI calls in and out of strings should Just Work, and quite possibly would work without incurring an extra conversion. Nice work if you can get it, but there isn't a UTF-16 codec in pg, so it ends up the same place as class II. IV. Use real UCS-4 consistently. That's the pg_wchar encoding and there are functions and macros provided by pg for converting between UTF-8 and wchar, either character-at-a-time or whole-string-at-a-time. In recent Javas, String does have a constructor from int[], and a method getCodePointAt you can call in a loop, but no int[] getCodePoints(). In pg there is even a table giving foo <-> wchar converters for other server encodings, so it could be possible to skip the intermediate UTF-8 step and go straight from server encoding to UCS-4 and build the string from that. What's the problem with that? The only coders you find in the table are the whole-string-at-a-time ones, and they *don't have any parameter for output buffer length*! You have to guess, worst-case expansion, and preallocate the whole thing. (Or allocate at the end of memory, and catch SIGSEGV....) And the worst-case expansion is a lot bigger than the average case. I've shifted my affections over the last week among all four of those classes, but I am currently leaning toward class I. (From some googling, that seems to be a common solution for other Java devs too.) I *am* interested in using the lower-level methods in the hope of at least slightly reducing the reallocating and copying involved. Class I could also become a class I' where you find out if the JVM knows a coding that corresponds directly to the server encoding, and skip the UTF-8 step. I am not planning to implement that, at least not at first. In the case of server encoding == utf-8, that essentially happens anyway, since the pg_do_encoding_conversion... makes itself a no-op in that case. I assume utf-8 is the usual server encoding these days, so for a first effort that sounds "good enough". It will still be correct for other encodings, just slower. -Chap