Subject: Re: real/float example for testlibpq3

From: Tom Lane <[email-masked]>
Date: Mon, 28 Feb 2022 10:19:00 -0500

Ashutosh Bapat <[email-masked]> writes:
> I am wondering whether we should provide PQgetfloatvalue() which does
> what that example shows but transparently. It will return a float
> value of the given field instead of char *.

I'm against that, precisely because ...

> That will standardize the
> way to fetch real typed values in libpq. That leads to the next
> question. Do we need to introduce different PQget*value() for standard
> C/SQL data types.

... I do not want to go here.  Where would you stop?  How would you
deal with cross-machine inconsistencies in integer widths?

			regards, tom lane

---

From: Chapman Flack <[email-masked]>
Date: Mon, 28 Feb 2022 17:30:11 -0500

> deal with cross-machine inconsistencies in integer widths?

This stimulates a question for me.

Not just libpq, but all drivers implemented in whatever language, if they
wish to support binary protocol, depend on knowing what the committed
send/recv wire formats are for whichever types they mean to support.

In the current state of affairs, what's considered the ur-source of that
information?

I have often seen those formats documented in code comments, usually above
the recv function in the .c file for a given adt.

Have we got any more, well, machine-readable collection of that knowledge?

Regards,
-Chap

---

From: Tom Lane <[email-masked]>
Date: Mon, 28 Feb 2022 17:48:50 -0500

> In the current state of affairs, what's considered the ur-source of that

The source code for the type's send/receive functions :-(.  One could
wish for something better, but no one has stepped up to produce such
documentation.

			regards, tom lane

---

From: Greg Stark <[email-masked]>
Date: Wed, 30 Mar 2022 13:16:37 -0400

> wish for something better, but no one has stepped up to produce such

Fwiw the client library I heard of attempting to have good binary mode
support was the Crystal language client
https://github.com/will/crystal-pg. I think he was aiming for full
coverage of the built-in data types. That might make a good reference
implementation to write up documentation from. He probably uncovered
some corner cases in development that one might not find from just
inspection of the server code.

-- 
greg

---

From: Mark Wong <[email-masked]>
Date: Tue, 14 Jun 2022 10:40:06 -0700

> inspection of the server code.

Checking in for quick feedback to see if this refactor makes sense.

I've created a function for each data type with the idea that an example
for handling a specific data type can be more easily reviewed by looking
in a single place.

I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
this is a good way to go.

Regards,
Mark
diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c
index 4f7b791388..68972f17f3 100644
--- a/src/test/examples/testlibpq3.c
+++ b/src/test/examples/testlibpq3.c
@@ -11,21 +11,56 @@
  * CREATE SCHEMA testlibpq3;
  * SET search_path = testlibpq3;
  * SET standard_conforming_strings = ON;
- * CREATE TABLE test1 (i int4, t text, b bytea);
- * INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
- * INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
+ * CREATE SCHEMA testlibpq3;
+ * SET search_path = testlibpq3;
+ * SET standard_conforming_strings = ON;
+ * CREATE TABLE test1 (
+ *     i int4
+ *   , r real
+ *   , bo boolean
+ *   , ts timestamp
+ *   , t text
+ *   , b bytea
+ * );
+ * INSERT INTO test1
+ * VALUES (
+ *     1
+ *   , 3.141593
+ *   , true
+ *   , '2000-01-01 00:00:02.414213'
+ *   , 'joe''s place'
+ *   , '\000\001\002\003\004'
+ * );
+ * INSERT INTO test1
+ * VALUES (
+ *     2
+ *   , 1.618033
+ *   , false
+ *   , '2000-01-01 00:00:01.465571'
+ *   , 'ho there'
+ *   , '\004\003\002\001\000'
+ * );
  *
  * The expected output is:
  *
  * tuple 0: got
  *	i = (4 bytes) 1
+ *	r = (4 bytes) 3.141593
+ *	bo = (1 bytes) 1
  *	t = (11 bytes) 'joe's place'
  *	b = (5 bytes) \000\001\002\003\004
  *
  * tuple 0: got
  *	i = (4 bytes) 2
+ *	r = (4 bytes) 1.618033
+ *	bo = (1 bytes) 0
  *	t = (8 bytes) 'ho there'
  *	b = (5 bytes) \004\003\002\001\000
+ *
+ * General notes about this example:
+ *
+ * Use PQfnumber to avoid assumptions about field order in result but when
+ * getting the field values we ignore possibility they are null!
  */
 
 #ifdef WIN32
@@ -36,6 +71,7 @@
 #include <stdlib.h>
 #include <stdint.h>
 #include <string.h>
+#include <time.h>
 #include <sys/types.h>
 #include "libpq-fe.h"
 
@@ -44,6 +80,13 @@
 #include <arpa/inet.h>

+/* These macros hopefully make reading calculations for timestamps easier. */
+#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
+#define SECS_PER_DAY 86400
+
+uint64_t ntohll(uint64_t);
+
 static void
 exit_nicely(PGconn *conn)
 {
@@ -51,6 +94,142 @@ exit_nicely(PGconn *conn)
 	exit(1);
 }
 
+static void
+handle_boolean(PGresult *res, int i)
+{
+	int			fnum;
+	char	   *ptr;
+	int			val;
+
+	fnum = PQfnumber(res, "bo");
+	ptr = PQgetvalue(res, i, fnum);
+	val = (int) *ptr;
+	printf(" bo = (%d bytes) %d\n", PQgetlength(res, i, fnum), val);
+}
+
+static void
+handle_bytea(PGresult *res, int i)
+{
+	int			j;
+	int			fnum;
+	char	   *ptr;
+	int			len;
+
+	fnum = PQfnumber(res, "b");
+	ptr = PQgetvalue(res, i, fnum);
+
+	/*
+	 * The binary representation of BYTEA is a bunch of bytes, which could
+	 * include embedded nulls so we have to pay attention to field length.
+	 */
+	len = PQgetlength(res, i, fnum);
+	printf(" b = (%d bytes) ", len);
+	for (j = 0; j < len; j++) printf("\\%03o", ptr[j]);
+}
+
+static void
+handle_integer(PGresult *res, int i)
+{
+	int			fnum;
+	char	   *ptr;
+	int			val;
+
+	fnum = PQfnumber(res, "i");
+	ptr = PQgetvalue(res, i, fnum);
+
+	/*
+	 * The binary representation of INT4 is in network byte order, which
+	 * we'd better coerce to the local byte order.
+	 */
+	val = ntohl(*((uint32_t *) ptr));
+
+	printf(" i = (%d bytes) %d\n", PQgetlength(res, i, fnum), val);
+}
+
+static void
+handle_real(PGresult *res, int i)
+{
+	int			fnum;
+	char	   *ptr;
+	union {
+		int		i;
+		float	f;
+	}			val;
+
+	fnum = PQfnumber(res, "r");
+	ptr = PQgetvalue(res, i, fnum);
+
+	/*
+	 * The binary representation of INT4 is in network byte order, which
+	 * we'd better coerce to the local byte order.
+	 */
+	val.i = ntohl(*((uint32_t *) ptr));
+
+	printf(" r = (%d bytes) %f\n", PQgetlength(res, i, fnum), val.f);
+}
+
+static void
+handle_text(PGresult *res, int i)
+{
+	int			fnum;
+	char	   *ptr;
+
+	fnum = PQfnumber(res, "t");
+	ptr = PQgetvalue(res, i, fnum);
+
+	/*
+	 * The binary representation of TEXT is, well, text, and since libpq was
+	 * nice enough to append a zero byte to it, it'll work just fine as a C
+	 * string.
+	 */
+	printf(" t = (%d bytes) '%s'\n", PQgetlength(res, i, fnum), ptr);
+}
+
+static void
+handle_timestamp(PGresult *res, int i)
+{
+	int			fnum;
+	char	   *ptr;
+	uint64_t	val;
+
+	struct tm  *tm;
+	time_t		timep;
+	uint32_t	mantissa;
+
+	fnum = PQfnumber(res, "ts");
+	ptr = PQgetvalue(res, i, fnum);
+	val = ntohll(*((uint64_t *) ptr));
+
+	/*
+	 * The binary representation of a timestamp is in microseconds
+	 * from 2000-01-01.
+	 */
+	timep = val / (uint64_t) 1000000 +
+			(uint64_t) (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) *
+			(uint64_t) SECS_PER_DAY;
+	mantissa = val - (uint64_t) (timep -
+			(POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY) *
+			(uint64_t) 1000000;
+
+	/* For ease of testing, assume and print timestamps in GMT. */
+	tm = gmtime(&timep);
+
+	printf(" ts = (%d bytes) %04d-%02d-%02d %02d:%02d:%02d.%06d\n",
+			PQgetlength(res, i, fnum), tm->tm_year + 1900, tm->tm_mon + 1,
+			tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec, mantissa);
+}
+
+/* This is a uint64_t version of ntohl from arpa/inet.h. */
+uint64_t
+ntohll(uint64_t netlonglong)
+{
+	if (ntohl(1) == 1)
+		return netlonglong;
+	else
+		return (uint64_t) (ntohl((int) ((netlonglong << 32) >> 32))) << 32 |
+				(uint64_t) ntohl(((int) (netlonglong >> 32)));
+}
+
 /*
  * This function prints a query result that is a binary-format fetch from
  * a table defined as in the comment above.  We split it out because the
@@ -59,54 +238,17 @@ exit_nicely(PGconn *conn)
 static void
 show_binary_results(PGresult *res)
 {
-	int			i,
-				j;
-	int			i_fnum,
-				t_fnum,
-				b_fnum;
-
-	/* Use PQfnumber to avoid assumptions about field order in result */
-	i_fnum = PQfnumber(res, "i");
-	t_fnum = PQfnumber(res, "t");
-	b_fnum = PQfnumber(res, "b");
+	int			i;
 
 	for (i = 0; i < PQntuples(res); i++)
 	{
-		char	   *iptr;
-		char	   *tptr;
-		char	   *bptr;
-		int			blen;
-		int			ival;
-
-		/* Get the field values (we ignore possibility they are null!) */
-		iptr = PQgetvalue(res, i, i_fnum);
-		tptr = PQgetvalue(res, i, t_fnum);
-		bptr = PQgetvalue(res, i, b_fnum);
-
-		/*
-		 * The binary representation of INT4 is in network byte order, which
-		 * we'd better coerce to the local byte order.
-		 */
-		ival = ntohl(*((uint32_t *) iptr));
-
-		/*
-		 * The binary representation of TEXT is, well, text, and since libpq
-		 * was nice enough to append a zero byte to it, it'll work just fine
-		 * as a C string.
-		 *
-		 * The binary representation of BYTEA is a bunch of bytes, which could
-		 * include embedded nulls so we have to pay attention to field length.
-		 */
-		blen = PQgetlength(res, i, b_fnum);
-
 		printf("tuple %d: got\n", i);
-		printf(" i = (%d bytes) %d\n",
-			   PQgetlength(res, i, i_fnum), ival);
-		printf(" t = (%d bytes) '%s'\n",
-			   PQgetlength(res, i, t_fnum), tptr);
-		printf(" b = (%d bytes) ", blen);
-		for (j = 0; j < blen; j++)
-			printf("\\%03o", bptr[j]);
+		handle_integer(res, i);
+		handle_real(res, i);
+		handle_boolean(res, i);
+		handle_timestamp(res, i);
+		handle_text(res, i);
+		handle_bytea(res, i);
 		printf("\n\n");
 	}
 }
diff --git a/src/test/examples/testlibpq3.sql b/src/test/examples/testlibpq3.sql
index 35a95ca347..94cb2b97b0 100644
--- a/src/test/examples/testlibpq3.sql
+++ b/src/test/examples/testlibpq3.sql
@@ -1,6 +1,29 @@
 CREATE SCHEMA testlibpq3;
 SET search_path = testlibpq3;
 SET standard_conforming_strings = ON;
-CREATE TABLE test1 (i int4, t text, b bytea);
-INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
-INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
+CREATE TABLE test1 (
+    i int4
+  , r real
+  , bo boolean
+  , ts timestamp
+  , t text
+  , b bytea
+);
+INSERT INTO test1
+VALUES (
+    1
+  , 3.141593
+  , true
+  , '2000-01-01 00:00:02.414213'
+  , 'joe''s place'
+  , '\000\001\002\003\004'
+);
+INSERT INTO test1
+VALUES (
+    2
+  , 1.618033
+  , false
+  , '2000-01-01 00:00:01.465571'
+  , 'ho there'
+  , '\004\003\002\001\000'
+);

---

From: Robert Haas <[email-masked]>
Date: Thu, 16 Jun 2022 15:06:01 -0400

> this is a good way to go.

I think this could be a reasonable kind of thing to do, but I think
you left the "ts" output out of the example output in the comments,
and also your code's apparently not portable, because my compiler is
OK with testlibpq3 right now, but with your patch it emits lengthy
unhappy moaning, starting with:

testlibpq3.c:88:10: error: expected ')'
uint64_t ntohll(uint64_t);
         ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_endian.h:140:25:
note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^

Apparently macOS defines ntohll as a macro, and it's not amused by
your attempt to include a function prototype for it.

I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program.  But
doing this much seems OK to me.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Tom Lane <[email-masked]>
Date: Thu, 16 Jun 2022 15:41:50 -0400

> doing this much seems OK to me.

Yeah, "hiding impromptu documentation inside a test program" is what
this looks like, and I'm not sure that's a reasonable way to go.

(1) Who's going to think to look in src/test/examples/testlibpq3.c for
documentation of binary formats?

(2) The useful details are likely to get buried in notational and
portability concerns, as I think your build failure illustrates.

(3) I bet few if any packagers install these files, so that the new
info would be unavailable to many people.

I think some new appendix in the main SGML docs would be the appropriate
place if we want to provide real documentation.

			regards, tom lane

---

From: Mark Wong <[email-masked]>
Date: Thu, 16 Jun 2022 15:56:50 -0700

> place if we want to provide real documentation.

Ok, I'll leave the testlibpq3.c as it was then.  If it's worth keeping
any of those changes, then I can remove the timestamp example because of
the ntohll() portability since that is trivial.

I'll start a new appendix and share again when I have something to show.

Regards,
Mark

--
Mark Wong

---

From: Mark Wong <[email-masked]>
Date: Fri, 1 Jul 2022 10:18:20 -0700

> place if we want to provide real documentation.

Just wanted to do another quick check in before I go too far.  How do
does this start look?  Extending the libpq section with a code snippet
and description per data type?

Regards,
Mark
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 37ec3cb4e5..44c60223ee 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -8874,6 +8874,109 @@ int PQisthreadsafe();
   </para>
  </sect1>
 
+ <sect1 id="libpq-binary-format">
+  <title>Binary Format</title>
+  <para>
+    Various <application>libpq</application> functions support sending or
+    receiving data in binary format.  Data in binary format requires the
+    application to be handle any differences between the system and the network
+    byte order.
+  </para>
+
+  <para>
+    <xref linkend="libpq-example-3"/> demonstrates how to write a complete
+    program that uses <application>libpq</application> functions that request
+    data in binary foramt.  This example shows how to handle only a few data
+    types.  This section will detail the handling of as many of the data types
+    as possible.  See <xref linkend="datatype"/> for descriptions of each of the
+    built-in data types.
+  </para>
+
+  <sect2 id="libpq-binary-boolean">
+   <title><type>boolean</type></title>
+    <para>
+       A <type>boolean</type> is transmitted as single byte that, when cast to
+       an <literal>int</literal>, will be <literal>0</literal> for
+       <literal>false</literal> and <literal>1</literal> for
+       <literal>true</literal>.
+    </para>
+<programlisting>
+<![CDATA[
+int         value;
+
+ptr = PQgetvalue(res, row_number, column_number);
+value = (int) *ptr;
+printf("%d\n", value);
+]]>
+</programlisting>
+  </sect2>
+
+  <sect2 id="libpq-binary-real">
+   <title><type>real</type></title>
+    <para>
+      A <type>real</type> is composed of 4 bytes and needs to be handled
+      correctly for byte order.
+    </para>
+<programlisting>
+<![CDATA[
+union {
+    int     i;
+    float   f;
+}           value;
+
+ptr = PQgetvalue(res, row_number, column_number);
+val.i = ntohl(*((uint32_t *) ptr));
+printf("%f\n", value.f);
+]]>
+</programlisting>
+  </sect2>
+
+  <sect2 id="libpq-binary-timestamp-without-time-zone">
+   <title><type>timestamp without time zone</type></title>
+    <para>
+      A <type>timestamp without time zone</type> is a 64-bit data type
+      representing the number of microseconds since January 1, 2000.  It can be
+      converted into a broken-down time representation by converting the time
+      into seconds and saving the microseconds elsewhere.
+    </para>
+    <para>
+      Note that in C time is counted from January 1, 1970 so this discrepency
+      needs to be accounted for in addition to handling the network byte order.
+    </para>
+<programlisting>
+<![CDATA[
+#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
+#define SECS_PER_DAY 86400
+
+uint64_t    value;
+
+struct tm  *tm;
+time_t      timep;
+uint32_t    mantissa;
+
+ptr = PQgetvalue(res, column_number, row_number);
+/* Note ntohll() is not implemented on all platforms. */
+val = ntohll(*((uint64_t *) ptr));
+
+timep = val / (uint64_t) 1000000 +
+        (uint64_t) (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) *
+        (uint64_t) SECS_PER_DAY;
+mantissa = val - (uint64_t) (timep -
+        (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY) *
+        (uint64_t) 1000000;
+
+/* Assume and print timestamps in GMT for simplicity. */
+tm = gmtime(&timep);
+
+printf("%04d-%02d-%02d %02d:%02d:%02d.%06d\n",
+        tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday, tm->tm_hour,
+        tm->tm_min, tm->tm_sec, mantissa);
+]]>
+</programlisting>
+  </sect2>
+
+ </sect1>
 
  <sect1 id="libpq-build">
   <title>Building <application>libpq</application> Programs</title>

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 1 Nov 2022 11:39:40 +0100

> and description per data type?

If we want to start documenting the binary format for all data types, it 
should go into the "Data Types" chapter.  There, we already document the 
text format in great detail, so putting the binary format in there as 
well would make sense.

The libpq chapter could then contain some examples of applying that 
information in the context of libpq.  But the libpq documentation 
shouldn't be the primary place where the formats are documented.

---

From: Tom Lane <[email-masked]>
Date: Tue, 01 Nov 2022 09:15:46 -0400

> text format in great detail, so putting the binary format in there as 

Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types".  Perhaps we should
invent a new appendix or the like?  Somewhere near the wire protocol
docs seems sensible.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 3 Nov 2022 09:23:00 -0400

> invent a new appendix or the like?  Somewhere near the wire protocol

Would that clutter the protocol docs? ;-)

I suppose figuring out exactly where to put it and how to mark it up, 
etc., in a repeatable fashion is part of the job here.

---

From: Tom Lane <[email-masked]>
Date: Thu, 03 Nov 2022 09:55:22 -0400

> Would that clutter the protocol docs? ;-)

I said "near", not "in".  At the time I was thinking "new appendix",
but I now recall that the wire protocol docs are not an appendix
but a chapter in the Internals division.  So that doesn't seem like
quite the right place anyway.

Perhaps a new chapter under "IV. Client Interfaces" is the right
place?

If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.

> etc., in a repeatable fashion is part of the job here.

Yup.

			regards, tom lane

---

From: Mark Wong <[email-masked]>
Date: Wed, 9 Nov 2022 12:47:23 -0800

> > etc., in a repeatable fashion is part of the job here.

I'll take a stab at adding a new chapter and share how that looks.

Regards,
Mark

--
Mark Wong
EDB https://enterprisedb.com

---

From: Michael Paquier <[email-masked]>
Date: Wed, 30 Nov 2022 15:59:12 +0900

> I'll take a stab at adding a new chapter and share how that looks.

This thread has stalled for three weeks, so I have marked it as
returned with feedback.  Once you have a new patch, please feel free
to submit it.
--
Michael