Subject: [PROPOSAL]a new data type 'bytea' for ECPG

From: "Matsumura, Ryo" <[email-masked]>
Date: Mon, 1 Oct 2018 08:03:42 +0000

Hi, Hackers

# This is my first post.

I will try to implement a new data type 'bytea' for ECPG.
I think that the implementation is not complicated.
Does anyone need it ?

* Why do I need bytea ?

Currently, ECPG program can treat binary data for bytea column with 'char' type
of C language, but it must convert from/to escaped format with PQunescapeBytea/
PQescapeBytea(). It forces users to add an unnecessary code and to pay cost for
the conversion in runtime.
# My PoC will not be able to solve output conversion cost.

I think that set/put data for host variable should be more simple.
The following is an example of Oracle Pro *C program for RAW type column.

  VARCHAR   raw_data[20];

  /* preprocessed to the following
   * struct 
   * { 
   *    unsigned short  len; 
   *    unsigned char   arr[20]; 
   * } raw_data;
   */

  raw_data.len = 10;
  memcpy(raw_data.arr, data, 10);

  see also:
  https://docs.oracle.com/cd/E11882_01/appdev.112/e10825/pc_04dat.htm#i23305

In ECPG, varchar host variable cannot be used for bytea because it cannot treat
'\0' as part of data. If the length is set to 10 and there is '\0' at 3rd byte,
ecpglib truncates 3rd byte and later at the following:

  [src/interfaces/ecpg/ecpglib/execute.c]
  ecpg_store_input(const int lineno, const bool force_indicator, const struct
  :
      switch (var->type)
  :
        case ECPGt_varchar:
          if (!(newcopy = (char *) ecpg_alloc(variable->len + 1, lineno)))
            return false;
  !!      strncpy(newcopy, variable->arr, variable->len);
          newcopy[variable->len] = '\0';

I also think that the behavior of varchar host variable should not be changed
because of compatibility.
Therefore, a new type of host variable 'bytea' is needed.

Since ecpglib can distinguish between C string and binary data, it can send
binary data to backend directly by using 'paramFormats' argument of PQexecParams().
Unfortunately, the conversion of output data cannot be omitted in ecpglib because
libpq doesn't provide like 'paramFormats'.
 ('resultFormat' means that *all* data from backend is formatted by binary or not.)

  PQexecParams(PGconn *conn,
             const char *command,
             int nParams,
             const Oid *paramTypes,
             const char *const *paramValues,
             const int *paramLengths,
  !!         const int *paramFormats,
             int resultFormat)

* How to use new 'bytea' ?

ECPG programmers can use almost same as 'varchar' but cannot use as names.
(e.g. connection name, prepared statement name, cursor name and so on)

 - Can use in Declare Section.

  exec sql begin declare section;
    bytea data1[512];
    bytea data2[DATA_SIZE];   /* can use macro */
    bytea send_data[DATA_NUM][DATA_SIZE];  /* can use two dimensional array */
    bytea recv_data[][DATA_SIZE]; /* can use flexible array */
  exec sql end declare section;

 - Can *not* use for name.

  exec sql begin declare section;
    bytea conn_name[DATA_SIZE];
  exec sql end declare section;

  exec sql connect to :conn_name;   !! error

 - Conversion is not needed in user program.

  exec sql begin declare section;
      bytea send_buf[DATA_SIZE];
      bytea recv_buf[DATA_SIZE - 13];
      int ind_recv;
  exec sql end declare section;

  exec sql create table test (data1 bytea);
  exec sql truncate test;
  exec sql insert into test (data1) values (:send_buf);
  exec sql select data1 into :recv_buf:ind_recv from test;
  /* ind_recv is set to 13. */

* How to preprocess 'bytea' ?

  'bytea' is preprocessed almost same as varchar.
  The following is preprocessed to the next.

    exec sql begin declare section;
      bytea data[DATA_SIZE];
      bytea send_data[DATA_NUM][DATA_SIZE];
      bytea recv_data[][DATA_SIZE];
    exec sql end declare section;

    struct bytea_1 {int len; char arr[DATA_SIZE]} data; 
    struct bytea_2 {int len; char arr[DATA_SIZE]} send_data[DATA_NUM]; 
    struct bytea_3 {int len; char arr[DATA_SIZE]} *recv_data;

Thank you for your consideration.

Regards
Ryo Matsumura

---

From: "Matsumura, Ryo" <[email-masked]>
Date: Wed, 24 Oct 2018 09:43:31 +0000

Hackers

No one commented to the proposal, but I'm not discouraged.
I attach a patch. Please review or comment to proposal.

Note:
- The patch cannot yet decode escape format data from backend.
- [ecpg/test/expected/sql-bytea.stderr] in patch includes non-ascii data.

I explain a little about the patch.

Preproc:
  Almost same as varchar.

Ecpglib:
- ecpg_build_params()
  Build two more arrays paramlengths and paramformats for PQexecParams().
  If input variable type is bytea, set pramformats to 1(= is binary) and
  set binary data length to paramlengths.

- ecpg_store_input()
  If input variable type is bytea, copy its binary data to ecpg_alloc-ed area directly.

- ecpg_get_data()
  If output variable type is bytea, decode received results to user area.
  Encode/decode function is imported from backend/utils/adt/encode.c

- ECPGset_desc()
  Currently ecpglib saves data to internal area(struct descriptor_item) for execution phase,
  but doesn't save type information that is needed in case of bytea.
  So I add a member is_binary to descriptor_item structure.

Thank you.

Regards
Ryo Matsumura

> -----Original Message-----
> From: Matsumura, Ryo [mailto:[email-masked]]
> Sent: Monday, October 1, 2018 5:04 PM
> To: [email-masked]
> Subject: [PROPOSAL]a new data type 'bytea' for ECPG
> 
> Hi, Hackers
> 
> # This is my first post.
>

---

From: Michael Meskes <[email-masked]>
Date: Wed, 24 Oct 2018 15:29:12 +0200

Hi Matsumoro-san,

thanks for your effort and apologies for the late reply. 

> The following is an example of Oracle Pro *C program for RAW type

Just to be clear, Oracle can use varchars for binary data, right?

> ecpglib truncates 3rd byte and later at the following:

I've been traveling too much to check, but does the standard say
anything about that?

> Therefore, a new type of host variable 'bytea' is needed.

This I am actually not sure about. I think we should try to stick with
the standard and, if it does not comment on it, with what others in the
market do to make migrations easier. So far I do not remember any
database having a bytea datatype in embedded SQL.

Comments anyone?

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

---

From: "Matsumura, Ryo" <[email-masked]>
Date: Thu, 25 Oct 2018 09:33:12 +0000

Hi Michael

Thank you for your comment.

# Please call me Ryo. Matsumura is too long.

> Just to be clear, Oracle can use varchars for binary data, right?

I'm sorry. That is my mistake.
In Pro*C, the data should be represented as hex format C string.

> I've been traveling too much to check, but does the standard say

bytea as a type of table definition may correspond to BLOB in the standard.
If it is true, the standard defines corresponding type in C as the following:

  ------
  struct {
    long hvn_reserved
    unsigned long hvn_length
    char hvn_data[L];
  } hvn

  * hvn is the name of the host variable defined to correspond
    to the SQL data type
  ------

> database having a bytea datatype in embedded SQL.

Maybe Oracle doesn't have it.
I found documents of DB2.

  blob(n) correspond to BLOB in V11.
  https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.apdv.embed.doc/doc/r0006090.html

  Since I don't have DB2 installation, I cannot confirm typedef of blob(n).
  But in V9 the following structure correspond to BLOB.
  https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.7.0/com.ibm.db2.luw.apdv.routines.doc/doc/c0009190.html

  struct sqludf_lob
  {
    sqluint32     length;      /* length in bytes */
    char          data[1];      /* first byte of lob */ 
  }; 

It seems that there is no defact and no product following to the standards.
I wonder whether bytea should follow to the standard completely or
follow to existing varchar for usability.

Thank you.

Regards
Ryo Matsumura

---

From: Michael Meskes <[email-masked]>
Date: Fri, 26 Oct 2018 20:43:18 +0200

Hi Ryo-san,

> # Please call me Ryo. Matsumura is too long.

Thanks.

> In Pro*C, the data should be represented as hex format C string.

Just to clarify, there is no special datatype for binary data?

> bytea as a type of table definition may correspond to BLOB in the

Would we prefer to add a blob datatype then?

> follow to existing varchar for usability.

Do you see any disadvantage of following the standard? I don't really
see where the usability drawback is. In general I would prefer being as
close to the standard as reasonably possible.

Michael

---

From: "Tsunakawa, Takayuki" <[email-masked]>
Date: Tue, 30 Oct 2018 06:24:48 +0000

From: Michael Meskes [mailto:[email-masked]]
> close to the standard as reasonably possible.

I think the host variable data type that corresponds to the server-side bytea should be bytea.  As the following pages state or imply, it would be better to create standard-compliant LOB types someday, and use the keyword BLOB in ECPG for that type.  The server-side data types should have the names BLOB, CLOB and NCLOB.  Those types should handle data larget than 1 GB and have the locator feature defined in the SQL standard.  Maybe we should also advanced LOB features like Oracle's SecureFiles LOB and SQL Server's FileTables.

https://www.postgresql.org/docs/current/static/datatype-binary.html

"The SQL standard defines a different binary string type, called BLOB or BINARY LARGE OBJECT. The input format is different from bytea, but the provided functions and operators are mostly the same."

BinaryFilesInDB
https://wiki.postgresql.org/wiki/BinaryFilesInDB

Regards
Takayuki Tsunakawa

---

From: "Matsumura, Ryo" <[email-masked]>
Date: Tue, 30 Oct 2018 09:46:48 +0000

Hi Michael

> Just to clarify, there is no special datatype for binary data?

I apology for lack of research again.
Since it's a little difficult to answer, I explain by samples.

The following works.

  unsigned char buffer[128]; /* It's not needed to declare in DECLARE section. */
  exec sql var buffer is raw(128); /* This sematics may be different in ECPG. */
  exec sql create table test(c1 raw(128));
  exec sql insert into test(c1) values(:buffer);

The following cannot be pre-compiled.
In this sence, there is no special datatype for binary data.

  exec sql begin declare section;
  raw buffer[128]; /* error */
  exec sql end declare section;

  exec sql create table test(c1 raw(128));
  exec sql insert into test(c1) values(:buffer);

> Would we prefer to add a blob datatype then?

I think that blob datatype is needed for large binary data *in finally*,
but blob datatype and its access methods(*) is not needed for non-large
binary data(e.g. use for text data of which encoding is different from
database encoding) because of its complexity.
My proposal is mainly for non-large binary data.

(*) e.g. In Pro*C, OPEN, READ, WRITE, CLOSE, APPEND, and so on.

> close to the standard as reasonably possible.

I think there is no special reason to follow to existing varchar.
I can accept the standard. (Re-implementation is not difficult.)

Thank you.

Regards
Ryo Matsumura

> Sent: Saturday, October 27, 2018 3:43 AM
> To: Matsumura, Ryo/松村 量 <[email-masked]>;
> [email-masked]
> Subject: Re: [PROPOSAL]a new data type 'bytea' for ECPG
> 
> Hi Ryo-san,
>