agora inbox for pljava-dev@postgresql.org
help / color / mirror / Atom feed[Pljava-dev] UDT send and receive
6+ messages / 0 participants
[nested] [flat]
* [Pljava-dev] UDT send and receive
@ 2006-09-25 15:05
2006-09-25 15:58 ` [Pljava-dev] UDT send and receive
0 siblings, 1 reply; 6+ messages in thread
From: @ 2006-09-25 15:05 UTC (permalink / raw)
Hi, Thomas,
Markus Schaber wrote:
> I'm definitely planning to look into the C code. However, I might need
> some time to understand the "inner workings", and there's some higher
> priority work on my table, so it might take a few days.
I'm currently stuck in the C code in UDT.h.
From looking at the code, it seems to assume that the send and receive
code assumes that the internal (on-disk) representation is the same than
the one used for binary I/O, rather than relying on the send and receive
functions. Also, there are CREATE FUNCTION calls in the example on
http://wiki.tada.se/wiki/display/pljava/Creating+a+Scalar+UDT+in+Java
but the java code does not actually define them.
Is that assumption correct?
At least PostGIS currently uses a slightly different internal
representation internally compared to what send/receive use. The
internal on-disk format is optimized, the external representation is an
upwards compatible extension to the OpenGIS standardized WKB format.
So, from my understanding of the code, it's currently impossible to
implement an 1:1 replacement for PostGIS in pljava. (Not that I
seriously plan to do this.)
Btw, I have the impression that I'm the first one actually trying VARLEN
UDT mapping with pljava :-)
Thanks,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS
Fight against software patents in Europe! www.ffii.org
www.nosoftwarepatents.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <http://lists.pgfoundry.org/pipermail/pljava-dev/attachments/20060925/99e56583/attachment.bin;
^ permalink raw reply [nested|flat] 6+ messages in thread
* [Pljava-dev] UDT send and receive
2006-09-25 15:05 [Pljava-dev] UDT send and receive
@ 2006-09-25 15:58 `
2006-09-25 17:45 ` [Pljava-dev] UDT send and receive
0 siblings, 1 reply; 6+ messages in thread
From: @ 2006-09-25 15:58 UTC (permalink / raw)
Markus Schaber wrote:
> Hi, Thomas,
>
> Markus Schaber wrote:
>
>
>> I'm definitely planning to look into the C code. However, I might need
>> some time to understand the "inner workings", and there's some higher
>> priority work on my table, so it might take a few days.
>>
>
> I'm currently stuck in the C code in UDT.h.
>
> From looking at the code, it seems to assume that the send and receive
> code assumes that the internal (on-disk) representation is the same than
> the one used for binary I/O, rather than relying on the send and receive
> functions.
Not sure what you mean. The UDT functions *are* the send/receive
functions (with an added UDT parameter). They don't care much about the
representation as such. There's a length (-2, -1 or a verbatim length)
and then there are bytes of data. The send fucntion uses byteasend,
unknownsend, or a StringInfo depending on the length. The receive
performs the corresponding read. Totally representation agnostic.
> Also, there are CREATE FUNCTION calls in the example on
> http://wiki.tada.se/wiki/display/pljava/Creating+a+Scalar+UDT+in+Java
> but the java code does not actually define them.
>
> Is that assumption correct?
>
>
Yes. They will all be redirected to the UDT_input, UDT_output, UDT_send,
and UDT_receive that you'll find in UTD.c
> At least PostGIS currently uses a slightly different internal
> representation internally compared to what send/receive use. The
> internal on-disk format is optimized, the external representation is an
> upwards compatible extension to the OpenGIS standardized WKB format.
>
OK, so if you want to read that in Java, I assume your SQLData
input/outpout methods must deal with that. Perhaps I miss the point
altogether here. Who converts between the internal format and the
on-disk format?
> So, from my understanding of the code, it's currently impossible to
> implement an 1:1 replacement for PostGIS in pljava. (Not that I
> seriously plan to do this.)
>
>
Not sure I understand why. Why can the data conversion not take place in
Java, should you choose to do that?
> Btw, I have the impression that I'm the first one actually trying VARLEN
> UDT mapping with pljava :-)
>
>
You are. As you've discovered, it's broken.
Regards,
Thomas Hallgren
^ permalink raw reply [nested|flat] 6+ messages in thread
* [Pljava-dev] UDT send and receive
2006-09-25 15:05 [Pljava-dev] UDT send and receive
2006-09-25 15:58 ` [Pljava-dev] UDT send and receive
@ 2006-09-25 17:45 `
2006-09-25 18:58 ` [Pljava-dev] UDT send and receive
0 siblings, 1 reply; 6+ messages in thread
From: @ 2006-09-25 17:45 UTC (permalink / raw)
Hi, Thomas,
Thomas Hallgren wrote:
>> From looking at the code, it seems to assume that the send and receive
>> code assumes that the internal (on-disk) representation is the same than
>> the one used for binary I/O, rather than relying on the send and receive
>> functions.
> Not sure what you mean. The UDT functions *are* the send/receive
> functions (with an added UDT parameter). They don't care much about the
> representation as such. There's a length (-2, -1 or a verbatim length)
> and then there are bytes of data. The send fucntion uses byteasend,
> unknownsend, or a StringInfo depending on the length. The receive
> performs the corresponding read. Totally representation agnostic.
Yes. And, as I said, they send the on-disk data 1:1.
PostGIS internally uses a different storage format compare to binary
input/output, for abstraction purposes, and the send and receive
functions perform some conversions (including adding varlena header, and
adopting byte order).
In my eyes, your current approach will fail when someone gets a binary
dump of the data via COPY, and then reloads the data on a platform with
different endianness. And clients using binary V3 protocol need to know
the server's endianness.
>> At least PostGIS currently uses a slightly different internal
>> representation internally compared to what send/receive use. The
>> internal on-disk format is optimized, the external representation is an
>> upwards compatible extension to the OpenGIS standardized WKB format.
>>
> OK, so if you want to read that in Java, I assume your SQLData
> input/outpout methods must deal with that. Perhaps I miss the point
> altogether here. Who converts between the internal format and the
> on-disk format?
For PostGIS: the send and receive functions which are given at CREATE
DATATYPE.
http://svn.refractions.net/postgis/trunk/lwgeom/lwgeom_inout.c has the C
code for those functions, LWGEOM_recv() and LWGEOM_send() which call the
conversion routines WKBFromLWGEOM and LWGEOMFromWKB under the hood.
>> So, from my understanding of the code, it's currently impossible to
>> implement an 1:1 replacement for PostGIS in pljava. (Not that I
>> seriously plan to do this.)
>
> Not sure I understand why. Why can the data conversion not take place in
> Java, should you choose to do that?
Because there is nobody that calls the appropriate Java functions.
When an SQL COPY reads binary data, it is passed through the RECEIVE
function in UDT.c, and then put on the platters.
I hope that I managed to explain the problem, if not, please ask.
Thanks,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS
Fight against software patents in Europe! www.ffii.org
www.nosoftwarepatents.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <http://lists.pgfoundry.org/pipermail/pljava-dev/attachments/20060925/33eaa75f/attachment.bin;
^ permalink raw reply [nested|flat] 6+ messages in thread
* [Pljava-dev] UDT send and receive
2006-09-25 15:05 [Pljava-dev] UDT send and receive
2006-09-25 15:58 ` [Pljava-dev] UDT send and receive
2006-09-25 17:45 ` [Pljava-dev] UDT send and receive
@ 2006-09-25 18:58 `
2006-09-25 19:51 ` [Pljava-dev] UDT send and receive
0 siblings, 1 reply; 6+ messages in thread
From: @ 2006-09-25 18:58 UTC (permalink / raw)
Markus Schaber wrote:
> Hi, Thomas,
>
> Thomas Hallgren wrote:
>
>
>>> From looking at the code, it seems to assume that the send and receive
>>> code assumes that the internal (on-disk) representation is the same than
>>> the one used for binary I/O, rather than relying on the send and receive
>>> functions.
>>>
>> Not sure what you mean. The UDT functions *are* the send/receive
>> functions (with an added UDT parameter). They don't care much about the
>> representation as such. There's a length (-2, -1 or a verbatim length)
>> and then there are bytes of data. The send fucntion uses byteasend,
>> unknownsend, or a StringInfo depending on the length. The receive
>> performs the corresponding read. Totally representation agnostic.
>>
>
> Yes. And, as I said, they send the on-disk data 1:1.
>
> PostGIS internally uses a different storage format compare to binary
> input/output, for abstraction purposes, and the send and receive
> functions perform some conversions (including adding varlena header, and
> adopting byte order).
>
> In my eyes, your current approach will fail when someone gets a binary
> dump of the data via COPY, and then reloads the data on a platform with
> different endianness. And clients using binary V3 protocol need to know
> the server's endianness.
>
>
We're talking past each other here. The send/receive functions are the
ones that perform the conversion. If you do it in Java, you write
send/receive functions in Java. What you see is just the middle man,
passing data to/from such functions. It pays no specific attention to
representation. That is something you must do yourself in the
corresponding methods of the SQLData implementation.
>>> At least PostGIS currently uses a slightly different internal
>>> representation internally compared to what send/receive use. The
>>> internal on-disk format is optimized, the external representation is an
>>> upwards compatible extension to the OpenGIS standardized WKB format.
>>>
>>>
>> OK, so if you want to read that in Java, I assume your SQLData
>> input/outpout methods must deal with that. Perhaps I miss the point
>> altogether here. Who converts between the internal format and the
>> on-disk format?
>>
>
> For PostGIS: the send and receive functions which are given at CREATE
> DATATYPE.
>
>
Well, yes. But if you want to use the data types in you'll need to have
corresponding functions there. Are you suggesting that when reading, the
receive in the CREATE DATATYPE should be called first, and then, another
function should be called that would create the actual Java object from
the result of what the receive would output? IMHO, that's just pushing
the problem one step ahead. The SQLData implementation might just as
well mimic what the receive does in the first place, i.e. why not manage
the input to the receive rather then the output and save one roundtrip?
> http://svn.refractions.net/postgis/trunk/lwgeom/lwgeom_inout.c has the C
> code for those functions, LWGEOM_recv() and LWGEOM_send() which call the
> conversion routines WKBFromLWGEOM and LWGEOMFromWKB under the hood.
>
>
OK, so it would be fully possible to create special purpose Java/C JNI
mappings for that. But what we are discussing here is a *generic* way to
map virtually everything. You can do that by mimicing what lwgeom_inout
is doing in Java. If you don't want to mimic that, and if you don't want
special purpose JNI functions, then I have a hard time understanding how
you'd go about doing the mapping.
>
>>> So, from my understanding of the code, it's currently impossible to
>>> implement an 1:1 replacement for PostGIS in pljava. (Not that I
>>> seriously plan to do this.)
>>>
>> Not sure I understand why. Why can the data conversion not take place in
>> Java, should you choose to do that?
>>
>
> Because there is nobody that calls the appropriate Java functions.
>
>
On the contrary. If you map a PostGIS datatype to a SQLData
implementation class, PL/Java will see to that an instance of that class
is created and that it is fed with raw data from the PostGIS data type.
> When an SQL COPY reads binary data, it is passed through the RECEIVE
> function in UDT.c, and then put on the platters.
>
>
Right, and in that case you never see Java do anything. Not unless the
type is a UDT that is fully Java of course (with a CREATE DATATYPE that
actually appoints the functions in UDT.c)
> I hope that I managed to explain the problem, if not, please ask.
>
>
Still unclear I'm afraid :-)
Regards,
Thomas Hallgren
^ permalink raw reply [nested|flat] 6+ messages in thread
* [Pljava-dev] UDT send and receive
2006-09-25 15:05 [Pljava-dev] UDT send and receive
2006-09-25 15:58 ` [Pljava-dev] UDT send and receive
2006-09-25 17:45 ` [Pljava-dev] UDT send and receive
2006-09-25 18:58 ` [Pljava-dev] UDT send and receive
@ 2006-09-25 19:51 `
2006-09-25 20:47 ` [Pljava-dev] UDT send and receive
0 siblings, 1 reply; 6+ messages in thread
From: @ 2006-09-25 19:51 UTC (permalink / raw)
Hi, Thomas,
[Foreword: Currently, I don't have any intent of defining the PostGIS
datatype as a full pljava-defined UDT (the C code deals fine with this).
A "simple" mapping (and thus the readSQL/writeSQL methods) are enough
for me now. I just want to ensure that my understanding of the whole
issue is correct, and that we remove all limitations that would prevent
others (or me in the future) from implementing full UDT mappings.]
Thomas Hallgren wrote:
> We're talking past each other here. The send/receive functions are the
> ones that perform the conversion. If you do it in Java, you write
> send/receive functions in Java. What you see is just the middle man,
> passing data to/from such functions. It pays no specific attention to
> representation. That is something you must do yourself in the
> corresponding methods of the SQLData implementation.
Ah, now I understand.
I erroneously assumed the same Magic like the input/output functions
mapping to parse()/toString().
You suggest to implement send and receive as normal static functions in
Java, and then use CREATE FUNCTION without the "UDT[foo] output" special
syntax.
The Problem I see here is that, for the receive function, the Datatype
"internal" is used as function parameter, which currently has no mapping
for pljava, at least according to
http://wiki.tada.se/wiki/display/pljava/Default+Type+Mapping
So I can define the send function (which returns bytea) the way you
suggest, but not the receive function.
> Well, yes. But if you want to use the data types in you'll need to have
> corresponding functions there. Are you suggesting that when reading, the
> receive in the CREATE DATATYPE should be called first, and then, another
> function should be called that would create the actual Java object from
> the result of what the receive would output?
Yes, at least that's how I understand PostgreSQL under the hoods.
Let me clarify my view of the things:
We have (at least) 4 different representations:
A) the "canonical text representation"
B) the "canonical binary representation"
C) the "internal" representation
D) Java Objects.
C) is what PostgreSQL passes around (to PostGIS C functions as well as
to the PLJava glue code), and stores on disk. The size is defined as
"internallength" in the datatype, and contained in a 4-byte VARLEN
header for variable length datatypes which have internallength set to
-1. (let's ignore TOAST and 0-terminated Strings for simplification.)
D) is what's passed around in "user functions" in pljava lands, obviously.
A) is used in psql, pg_dump, non-binary COPY, the V2 protocol and the
text mode of the V3 protocol.
B) is used in binary COPY and the binary mode of the V3 protocol.
The pljava UDT mapping converts between C and D via the readSQL() and
writeSQL() methods.
PostgreSQL uses the input and output functions defined for the type to
convert between A and C.
The send and receive functions for the datatype convert between B and C.
http://www.postgresql.org/docs/8.1/interactive/xtypes.html contains an
example with some less-sophisticated, but explicitly coded send and
receive functions.
I hope it is understandable what I try to explain.
Thanks for your patience,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS
Fight against software patents in Europe! www.ffii.org
www.nosoftwarepatents.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <http://lists.pgfoundry.org/pipermail/pljava-dev/attachments/20060925/8b4194bd/attachment.bin;
^ permalink raw reply [nested|flat] 6+ messages in thread
* [Pljava-dev] UDT send and receive
2006-09-25 15:05 [Pljava-dev] UDT send and receive
2006-09-25 15:58 ` [Pljava-dev] UDT send and receive
2006-09-25 17:45 ` [Pljava-dev] UDT send and receive
2006-09-25 18:58 ` [Pljava-dev] UDT send and receive
2006-09-25 19:51 ` [Pljava-dev] UDT send and receive
@ 2006-09-25 20:47 `
0 siblings, 0 replies; 6+ messages in thread
From: @ 2006-09-25 20:47 UTC (permalink / raw)
Markus Schaber wrote:
> I erroneously assumed the same Magic like the input/output functions
> mapping to parse()/toString().
>
Nope, no magic :-)
> You suggest to implement send and receive as normal static functions in
> Java, and then use CREATE FUNCTION without the "UDT[foo] output" special
> syntax.
>
Unless you already have a type with send/receive written in C, then yes.
If you want to map a pre-existing type, you can still do that, but then
you need to duplicate the send/receive functionality in Java in order to
access the internals of that type.
> The Problem I see here is that, for the receive function, the Datatype
> "internal" is used as function parameter, which currently has no mapping
> for pljava, at least according to
> http://wiki.tada.se/wiki/display/pljava/Default+Type+Mapping
>
In some sense, that's true. The mapping is in itself internal. It's
there though...
> So I can define the send function (which returns bytea) the way you
> suggest, but not the receive function.
>
Yes you can. The internal receive function will map the internal bytea
that it receives to an SQLInput (the SQLInputFromChunk) and will pass it
to a SQLData implementation (which the user must provide).
> We have (at least) 4 different representations:
>
> A) the "canonical text representation"
> B) the "canonical binary representation"
> C) the "internal" representation
> D) Java Objects.
>
>
> C) is what PostgreSQL passes around (to PostGIS C functions as well as
> to the PLJava glue code), and stores on disk. The size is defined as
> "internallength" in the datatype, and contained in a 4-byte VARLEN
> header for variable length datatypes which have internallength set to
> -1. (let's ignore TOAST and 0-terminated Strings for simplification.)
>
> D) is what's passed around in "user functions" in pljava lands,
obviously.
>
> A) is used in psql, pg_dump, non-binary COPY, the V2 protocol and the
> text mode of the V3 protocol.
>
> B) is used in binary COPY and the binary mode of the V3 protocol.
>
>
> The pljava UDT mapping converts between C and D via the readSQL() and
> writeSQL() methods.
>
> PostgreSQL uses the input and output functions defined for the type to
> convert between A and C.
>
> The send and receive functions for the datatype convert between B and C.
>
> http://www.postgresql.org/docs/8.1/interactive/xtypes.html contains an
> example with some less-sophisticated, but explicitly coded send and
> receive functions.
>
> I hope it is understandable what I try to explain.
>
Yes, this makes sense to me. Do you still find this approach limiting or
is it in line with what you would like to have?
Regards,
Thomas Hallgren
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2006-09-25 20:47 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-09-25 15:05 [Pljava-dev] UDT send and receive
2006-09-25 15:58 `
2006-09-25 17:45 `
2006-09-25 18:58 `
2006-09-25 19:51 `
2006-09-25 20:47 `
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox