public inbox for [email protected]
help / color / mirror / Atom feedRe: libpq compression
52+ messages / 16 participants
[nested] [flat]
* Re: libpq compression
@ 2018-05-16 15:09 Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Grigory Smolkin @ 2018-05-16 15:09 UTC (permalink / raw)
To: [email protected]; +Cc: Konstantin Knizhnik <[email protected]>
Hello!
I have noticed that psql --help lack -Z|--compression option.
Also it would be nice to have option like --compression-level in psql
and pgbench.
On 03/30/2018 03:53 PM, Konstantin Knizhnik wrote:
> Hi hackers,
>
> One of our customers was managed to improve speed about 10 times by
> using SSL compression for the system where client and servers are
> located in different geographical regions
> and query results are very large because of JSON columns. Them
> actually do not need encryption, just compression.
> I expect that it is not the only case where compression of libpq
> protocol can be useful. Please notice that Postgres replication is
> also using libpq protocol.
>
> Taken in account that vulnerability was found in SSL compression and
> so SSLComppression is considered to be deprecated and insecure
> (http://www.postgresql-archive.org/disable-SSL-compression-td6010072.html),
> it will be nice to have some alternative mechanism of reducing libpq
> traffic.
>
> I have implemented some prototype implementation of it (patch is
> attached).
> To use zstd compression, Postgres should be configured with
> --with-zstd. Otherwise compression will use zlib unless it is disabled
> by --without-zlib option.
> I have added compression=on/off parameter to connection string and -Z
> option to psql and pgbench utilities.
> Below are some results:
>
> Compression ratio (raw->compressed):
>
>
> libz (level=1)
> libzstd (level=1)
> pgbench -i -s 10
> 16997209->2536330
> 16997209->268077
> pgbench -t 100000 -S
> 6289036->1523862
> 6600338<-900293
> 6288933->1777400
> 6600338<-1000318
>
>
> There is no mistyping: libzstd compress COPY data about 10 times
> better than libz, with wonderful compression ratio 63.
>
> Influence on execution time is minimal (I have tested local
> configuration when client and server are at the same host):
>
>
> no compression
> libz (level=1)
> libzstd (level=1)
> pgbench -i -s 10
> 1.552
> 1.572
> 1.611
> pgbench -t 100000 -S
> 4.482
> 4.926
> 4.877
>
> --
> Konstantin Knizhnik
> Postgres Professional:http://www.postgrespro.com
> The Russian Postgres Company
--
Grigory Smolkin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
@ 2018-05-16 15:54 ` Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-05-16 15:54 UTC (permalink / raw)
To: Grigory Smolkin <[email protected]>; [email protected]
On 16.05.2018 18:09, Grigory Smolkin wrote:
>
> Hello!
> I have noticed that psql --help lack -Z|--compression option.
> Also it would be nice to have option like --compression-level in psql
> and pgbench.
>
Thank you for this notice.
Updated and rebased patch is attached.
Concerning specification of compression level: I have made many
experiments with different data sets and both zlib/zstd and in both
cases using compression level higher than default doesn't cause some
noticeable increase of compression ratio, but quite significantly reduce
speed. Moreover, for "pgbench -i" zstd provides better compression ratio
(63 times!) with compression level 1 than with with largest recommended
compression level 22! This is why I decided not to allow user to choose
compression level.
Attachments:
[text/x-patch] libpq-compression-3.patch (35.2K, ../../[email protected]/2-libpq-compression-3.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..3a6f54b 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,31 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only conpression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression = 'z'; /* Request compression message */
+ int rc;
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, 1)) < 0 && errno == EINTR);
+ if (rc != 1)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +254,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +312,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +965,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +988,34 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("Failed to decompress data: %s", zpq_error(PqStream))));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1036,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1051,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1070,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1091,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1116,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1150,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1191,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1441,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1500,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f0c5149..efc4ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -188,6 +188,7 @@ int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
+bool libpq_compression; /* use libpq compression */
int main_pid; /* main process id used in log filename */
char *pghost = "";
@@ -590,6 +591,7 @@ usage(void)
" -h, --host=HOSTNAME database server host or socket directory\n"
" -p, --port=PORT database server port number\n"
" -U, --username=USERNAME connect as specified database user\n"
+ " -Z, --compression use libpq compression\n"
" -V, --version output version information, then exit\n"
" -?, --help show this help, then exit\n"
"\n"
@@ -1107,7 +1109,7 @@ doConnect(void)
*/
do
{
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
@@ -1124,8 +1126,10 @@ doConnect(void)
values[4] = dbName;
keywords[5] = "fallback_application_name";
values[5] = progname;
- keywords[6] = NULL;
- values[6] = NULL;
+ keywords[6] = "compression";
+ values[6] = libpq_compression ? "on" : "off";
+ keywords[7] = NULL;
+ values[7] = NULL;
new_pass = false;
@@ -4759,6 +4763,7 @@ main(int argc, char **argv)
{"builtin", required_argument, NULL, 'b'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
+ {"compression", no_argument, NULL, 'Z'},
{"debug", no_argument, NULL, 'd'},
{"define", required_argument, NULL, 'D'},
{"file", required_argument, NULL, 'f'},
@@ -4868,12 +4873,15 @@ main(int argc, char **argv)
exit(1);
}
- while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:Z", long_options, &optindex)) != -1)
{
char *script;
switch (c)
{
+ case 'Z':
+ libpq_compression = true;
+ break;
case 'i':
is_init_mode = true;
break;
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 702e742..ae7a14c 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -139,6 +139,7 @@ usage(unsigned short int pager)
fprintf(output, _(" -U, --username=USERNAME database user name (default: \"%s\")\n"), env);
fprintf(output, _(" -w, --no-password never prompt for password\n"));
fprintf(output, _(" -W, --password force password prompt (should happen automatically)\n"));
+ fprintf(output, _(" -Z, --compression compress traffic with server\n"));
fprintf(output, _("\nFor more information, type \"\\?\" (for internal commands) or \"\\help\" (for SQL\n"
"commands) from within psql, or consult the psql section in the PostgreSQL\n"
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index be57574..9271716 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -75,6 +75,7 @@ struct adhoc_opts
bool no_psqlrc;
bool single_txn;
bool list_dbs;
+ bool compression;
SimpleActionList actions;
};
@@ -237,8 +238,10 @@ main(int argc, char *argv[])
values[5] = pset.progname;
keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
- keywords[7] = NULL;
- values[7] = NULL;
+ keywords[7] = "compression";
+ values[7] = options.compression ? "on" : "off";
+ keywords[8] = NULL;
+ values[8] = NULL;
new_pass = false;
pset.db = PQconnectdbParams(keywords, values, true);
@@ -436,6 +439,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
{"echo-all", no_argument, NULL, 'a'},
{"no-align", no_argument, NULL, 'A'},
{"command", required_argument, NULL, 'c'},
+ {"compression", no_argument, NULL, 'Z'},
{"dbname", required_argument, NULL, 'd'},
{"echo-queries", no_argument, NULL, 'e'},
{"echo-errors", no_argument, NULL, 'b'},
@@ -476,7 +480,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
memset(options, 0, sizeof *options);
- while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01",
+ while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01Z",
long_options, &optindex)) != -1)
{
switch (c)
@@ -540,6 +544,9 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
case 'p':
options->port = pg_strdup(optarg);
break;
+ case 'Z':
+ options->compression = true;
+ break;
case 'P':
{
char *value;
diff --git a/src/bin/scripts/pg_isready.c b/src/bin/scripts/pg_isready.c
index f7ad7b4..5d035cd 100644
--- a/src/bin/scripts/pg_isready.c
+++ b/src/bin/scripts/pg_isready.c
@@ -34,7 +34,7 @@ main(int argc, char **argv)
const char *pghostaddr_str = NULL;
const char *pgport_str = NULL;
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..97326fd
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,338 @@
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void zpq_free(ZpqStream* zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const* zpq_error(ZpqStream* zs)
+{
+ return zs->rx_error;
+}
+
+size_t zpq_buffered(ZpqStream* zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ inflateInit(&zs->rx);
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void zpq_free(ZpqStream* zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const* zpq_error(ZpqStream* zs)
+{
+ return zs->rx.msg;
+}
+
+size_t zpq_buffered(ZpqStream* zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+#else
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg)
+{
+ return NULL;
+}
+
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size)
+{
+ return -1;
+}
+
+void zpq_free(ZpqStream* zs)
+{
+}
+
+char const* zpq_error(ZpqStream* zs)
+{
+ return NULL;
+}
+
+
+size_t zpq_buffered(ZpqStream* zs)
+{
+ return 0;
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..dc765af
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,28 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..e14c1f7 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,10 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..6fabc5b 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -269,6 +270,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
21, /* sizeof("tls-server-end-point") == 21 */
offsetof(struct pg_conn, scram_channel_binding)},
+ {"compression", "COMPRESSION", "0", NULL,
+ "ZSTD-Compression", "", 1,
+ offsetof(struct pg_conn, compression)},
+
/*
* ssl options are allowed even without client SSL support because the
* client can still handle SSL modes "disable" and "allow". Other
@@ -325,6 +330,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", NULL, NULL, NULL,
+ "Compression", "Z", 5,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +439,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2661,23 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3487,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-05 05:26 ` Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Thomas Munro @ 2018-06-05 05:26 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
<[email protected]> wrote:
> Thank you for this notice.
> Updated and rebased patch is attached.
Hi Konstantin,
Seems very useful. +1.
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
Does this actually guarantee that zs->rx.msg is set to a string? I
looked at some documentation here:
https://www.zlib.net/manual.html
It looks like return value Z_DATA_ERROR means that msg is set, but for
the other error codes Z_STREAM_ERROR, Z_BUF_ERROR, Z_MEM_ERROR it
doesn't explicitly say that. From a casual glance at
https://github.com/madler/zlib/blob/master/inflate.c I think it might
be set to Z_NULL and then never set to a string except in the mode =
BAD paths that produce the Z_DATA_ERROR return code. That's
interesting because later we do this:
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("Failed to decompress data: %s", zpq_error(PqStream))));
+ return EOF;
... where zpq_error() returns zs->rx.msg. That might crash or show
"(null)" depending on libc.
Also, message style: s/F/f/
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed)
Code style: We write "Type *foo", not "Type* var". We put the return
type of a function definition on its own line.
It looks like there is at least one place where zpq_stream.o's symbols
are needed but it isn't being linked in, so the build fails in some
ecpg stuff reached by make check-world:
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -fexcess-precision=standard -g -O2 -pthread -D_REENTRANT
-D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS test1.o
-L../../../../../src/port -L../../../../../src/common -L../../ecpglib
-lecpg -L../../pgtypeslib -lpgtypes
-L../../../../../src/interfaces/libpq -lpq -Wl,--as-needed
-Wl,-rpath,'/usr/local/pgsql/lib',--enable-new-dtags -lpgcommon
-lpgport -lpthread -lz -lreadline -lrt -lcrypt -ldl -lm -o test1
../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_free'
../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_error'
../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_read'
../../../../../src/interfaces/libpq/libpq.so: undefined reference to
`zpq_buffered'
../../../../../src/interfaces/libpq/libpq.so: undefined reference to
`zpq_create'
../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_write'
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
@ 2018-06-05 14:06 ` Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-05 14:06 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 05.06.2018 08:26, Thomas Munro wrote:
> On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
> <[email protected]> wrote:
>> Thank you for this notice.
>> Updated and rebased patch is attached.
> Hi Konstantin,
>
> Seems very useful. +1.
>
> + rc = inflate(&zs->rx, Z_SYNC_FLUSH);
> + if (rc != Z_OK)
> + {
> + return ZPQ_DECOMPRESS_ERROR;
> + }
>
> Does this actually guarantee that zs->rx.msg is set to a string? I
> looked at some documentation here:
>
> https://www.zlib.net/manual.html
>
> It looks like return value Z_DATA_ERROR means that msg is set, but for
> the other error codes Z_STREAM_ERROR, Z_BUF_ERROR, Z_MEM_ERROR it
> doesn't explicitly say that. From a casual glance at
> https://github.com/madler/zlib/blob/master/inflate.c I think it might
> be set to Z_NULL and then never set to a string except in the mode =
> BAD paths that produce the Z_DATA_ERROR return code. That's
> interesting because later we do this:
>
> + if (r == ZPQ_DECOMPRESS_ERROR)
> + {
> + ereport(COMMERROR,
> + (errcode_for_socket_access(),
> + errmsg("Failed to decompress data: %s", zpq_error(PqStream))));
> + return EOF;
>
> ... where zpq_error() returns zs->rx.msg. That might crash or show
> "(null)" depending on libc.
>
> Also, message style: s/F/f/
>
> +ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed)
>
> Code style: We write "Type *foo", not "Type* var". We put the return
> type of a function definition on its own line.
>
> It looks like there is at least one place where zpq_stream.o's symbols
> are needed but it isn't being linked in, so the build fails in some
> ecpg stuff reached by make check-world:
>
> gcc -Wall -Wmissing-prototypes -Wpointer-arith
> -Wdeclaration-after-statement -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -fexcess-precision=standard -g -O2 -pthread -D_REENTRANT
> -D_THREAD_SAFE -D_POSIX_PTHREAD_SEMANTICS test1.o
> -L../../../../../src/port -L../../../../../src/common -L../../ecpglib
> -lecpg -L../../pgtypeslib -lpgtypes
> -L../../../../../src/interfaces/libpq -lpq -Wl,--as-needed
> -Wl,-rpath,'/usr/local/pgsql/lib',--enable-new-dtags -lpgcommon
> -lpgport -lpthread -lz -lreadline -lrt -lcrypt -ldl -lm -o test1
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_free'
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_error'
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_read'
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to
> `zpq_buffered'
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to
> `zpq_create'
> ../../../../../src/interfaces/libpq/libpq.so: undefined reference to `zpq_write'
>
Hi Thomas,
Thank you for review. Updated version of the patch fixing all reported
problems is attached.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] libpq-compression-4.patch (37.1K, ../../[email protected]/2-libpq-compression-4.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..c963657 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,31 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only conpression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression = 'z'; /* Request compression message */
+ int rc;
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, 1)) < 0 && errno == EINTR);
+ if (rc != 1)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +254,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +312,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +965,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +988,37 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ char const* msg = zpq_error(PqStream);
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1039,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1054,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1073,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1094,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1119,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1153,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1194,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1444,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1503,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f0c5149..efc4ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -188,6 +188,7 @@ int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
+bool libpq_compression; /* use libpq compression */
int main_pid; /* main process id used in log filename */
char *pghost = "";
@@ -590,6 +591,7 @@ usage(void)
" -h, --host=HOSTNAME database server host or socket directory\n"
" -p, --port=PORT database server port number\n"
" -U, --username=USERNAME connect as specified database user\n"
+ " -Z, --compression use libpq compression\n"
" -V, --version output version information, then exit\n"
" -?, --help show this help, then exit\n"
"\n"
@@ -1107,7 +1109,7 @@ doConnect(void)
*/
do
{
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
@@ -1124,8 +1126,10 @@ doConnect(void)
values[4] = dbName;
keywords[5] = "fallback_application_name";
values[5] = progname;
- keywords[6] = NULL;
- values[6] = NULL;
+ keywords[6] = "compression";
+ values[6] = libpq_compression ? "on" : "off";
+ keywords[7] = NULL;
+ values[7] = NULL;
new_pass = false;
@@ -4759,6 +4763,7 @@ main(int argc, char **argv)
{"builtin", required_argument, NULL, 'b'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
+ {"compression", no_argument, NULL, 'Z'},
{"debug", no_argument, NULL, 'd'},
{"define", required_argument, NULL, 'D'},
{"file", required_argument, NULL, 'f'},
@@ -4868,12 +4873,15 @@ main(int argc, char **argv)
exit(1);
}
- while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:Z", long_options, &optindex)) != -1)
{
char *script;
switch (c)
{
+ case 'Z':
+ libpq_compression = true;
+ break;
case 'i':
is_init_mode = true;
break;
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 702e742..ae7a14c 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -139,6 +139,7 @@ usage(unsigned short int pager)
fprintf(output, _(" -U, --username=USERNAME database user name (default: \"%s\")\n"), env);
fprintf(output, _(" -w, --no-password never prompt for password\n"));
fprintf(output, _(" -W, --password force password prompt (should happen automatically)\n"));
+ fprintf(output, _(" -Z, --compression compress traffic with server\n"));
fprintf(output, _("\nFor more information, type \"\\?\" (for internal commands) or \"\\help\" (for SQL\n"
"commands) from within psql, or consult the psql section in the PostgreSQL\n"
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index be57574..9271716 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -75,6 +75,7 @@ struct adhoc_opts
bool no_psqlrc;
bool single_txn;
bool list_dbs;
+ bool compression;
SimpleActionList actions;
};
@@ -237,8 +238,10 @@ main(int argc, char *argv[])
values[5] = pset.progname;
keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
- keywords[7] = NULL;
- values[7] = NULL;
+ keywords[7] = "compression";
+ values[7] = options.compression ? "on" : "off";
+ keywords[8] = NULL;
+ values[8] = NULL;
new_pass = false;
pset.db = PQconnectdbParams(keywords, values, true);
@@ -436,6 +439,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
{"echo-all", no_argument, NULL, 'a'},
{"no-align", no_argument, NULL, 'A'},
{"command", required_argument, NULL, 'c'},
+ {"compression", no_argument, NULL, 'Z'},
{"dbname", required_argument, NULL, 'd'},
{"echo-queries", no_argument, NULL, 'e'},
{"echo-errors", no_argument, NULL, 'b'},
@@ -476,7 +480,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
memset(options, 0, sizeof *options);
- while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01",
+ while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01Z",
long_options, &optindex)) != -1)
{
switch (c)
@@ -540,6 +544,9 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
case 'p':
options->port = pg_strdup(optarg);
break;
+ case 'Z':
+ options->compression = true;
+ break;
case 'P':
{
char *value;
diff --git a/src/bin/scripts/pg_isready.c b/src/bin/scripts/pg_isready.c
index f7ad7b4..5d035cd 100644
--- a/src/bin/scripts/pg_isready.c
+++ b/src/bin/scripts/pg_isready.c
@@ -34,7 +34,7 @@ main(int argc, char **argv)
const char *pghostaddr_str = NULL;
const char *pgport_str = NULL;
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..1ddf82f
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,367 @@
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx_error;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ int rc;
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ rc = deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ rc = inflateInit(&zs->rx);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx.msg;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+#else
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ return NULL;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size)
+{
+ return -1;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return NULL;
+}
+
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return 0;
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..dc765af
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,28 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..2dad4fb 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,16 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
@@ -49,7 +59,7 @@ endif
# src/backend/utils/mb
OBJS += encnames.o wchar.o
# src/common
-OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o
+OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS += fe-secure-openssl.o fe-secure-common.o sha2_openssl.o
@@ -106,7 +116,7 @@ backend_src = $(top_srcdir)/src/backend
chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c: % : $(top_srcdir)/src/port/%
rm -f $@ && $(LN_S) $< .
-ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c: % : $(top_srcdir)/src/common/%
+ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c: % : $(top_srcdir)/src/common/%
rm -f $@ && $(LN_S) $< .
encnames.c wchar.c: % : $(backend_src)/utils/mb/%
@@ -156,7 +166,7 @@ clean distclean: clean-lib
rm -f pg_config_paths.h
# Remove files we (may have) symlinked in from src/port and other places
rm -f chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c
- rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c
+ rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c
rm -f encnames.c wchar.c
maintainer-clean: distclean maintainer-clean-lib
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..6fabc5b 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -269,6 +270,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
21, /* sizeof("tls-server-end-point") == 21 */
offsetof(struct pg_conn, scram_channel_binding)},
+ {"compression", "COMPRESSION", "0", NULL,
+ "ZSTD-Compression", "", 1,
+ offsetof(struct pg_conn, compression)},
+
/*
* ssl options are allowed even without client SSL support because the
* client can still handle SSL modes "disable" and "allow". Other
@@ -325,6 +330,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", NULL, NULL, NULL,
+ "Compression", "Z", 5,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +439,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2661,23 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3487,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-05 23:03 ` Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Thomas Munro @ 2018-06-05 23:03 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik
<[email protected]> wrote:
> Thank you for review. Updated version of the patch fixing all reported
> problems is attached.
Small problem on Windows[1]:
C:\projects\postgresql\src\include\common/zpq_stream.h(17): error
C2143: syntax error : missing ')' before '*'
[C:\projects\postgresql\libpq.vcxproj]
2395
You used ssize_t in zpq_stream.h, but Windows doesn't have that type.
We have our own typedef in win32_port.h. Perhaps zpq_stream.c should
include postgres.h/postgres_fe.h (depending on FRONTEND) like the
other .c files in src/common, before it includes zpq_stream.h?
Instead of "c.h".
[1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.1106
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
@ 2018-06-06 09:37 ` Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-06 09:37 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 06.06.2018 02:03, Thomas Munro wrote:
> On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik
> <[email protected]> wrote:
>> Thank you for review. Updated version of the patch fixing all reported
>> problems is attached.
> Small problem on Windows[1]:
>
> C:\projects\postgresql\src\include\common/zpq_stream.h(17): error
> C2143: syntax error : missing ')' before '*'
> [C:\projects\postgresql\libpq.vcxproj]
> 2395
>
> You used ssize_t in zpq_stream.h, but Windows doesn't have that type.
> We have our own typedef in win32_port.h. Perhaps zpq_stream.c should
> include postgres.h/postgres_fe.h (depending on FRONTEND) like the
> other .c files in src/common, before it includes zpq_stream.h?
> Instead of "c.h".
>
> [1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.1106
>
Thank you very much for reporting the problem.
I attached new patch with include of postgres_fe.h added to zpq_stream.c
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] libpq-compression-5.patch (37.2K, ../../[email protected]/2-libpq-compression-5.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..c963657 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,31 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only conpression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression = 'z'; /* Request compression message */
+ int rc;
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, 1)) < 0 && errno == EINTR);
+ if (rc != 1)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +254,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +312,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +965,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +988,37 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ char const* msg = zpq_error(PqStream);
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1039,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1054,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1073,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1094,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1119,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1153,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1194,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1444,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1503,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f0c5149..efc4ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -188,6 +188,7 @@ int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
+bool libpq_compression; /* use libpq compression */
int main_pid; /* main process id used in log filename */
char *pghost = "";
@@ -590,6 +591,7 @@ usage(void)
" -h, --host=HOSTNAME database server host or socket directory\n"
" -p, --port=PORT database server port number\n"
" -U, --username=USERNAME connect as specified database user\n"
+ " -Z, --compression use libpq compression\n"
" -V, --version output version information, then exit\n"
" -?, --help show this help, then exit\n"
"\n"
@@ -1107,7 +1109,7 @@ doConnect(void)
*/
do
{
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
@@ -1124,8 +1126,10 @@ doConnect(void)
values[4] = dbName;
keywords[5] = "fallback_application_name";
values[5] = progname;
- keywords[6] = NULL;
- values[6] = NULL;
+ keywords[6] = "compression";
+ values[6] = libpq_compression ? "on" : "off";
+ keywords[7] = NULL;
+ values[7] = NULL;
new_pass = false;
@@ -4759,6 +4763,7 @@ main(int argc, char **argv)
{"builtin", required_argument, NULL, 'b'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
+ {"compression", no_argument, NULL, 'Z'},
{"debug", no_argument, NULL, 'd'},
{"define", required_argument, NULL, 'D'},
{"file", required_argument, NULL, 'f'},
@@ -4868,12 +4873,15 @@ main(int argc, char **argv)
exit(1);
}
- while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:Z", long_options, &optindex)) != -1)
{
char *script;
switch (c)
{
+ case 'Z':
+ libpq_compression = true;
+ break;
case 'i':
is_init_mode = true;
break;
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 702e742..ae7a14c 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -139,6 +139,7 @@ usage(unsigned short int pager)
fprintf(output, _(" -U, --username=USERNAME database user name (default: \"%s\")\n"), env);
fprintf(output, _(" -w, --no-password never prompt for password\n"));
fprintf(output, _(" -W, --password force password prompt (should happen automatically)\n"));
+ fprintf(output, _(" -Z, --compression compress traffic with server\n"));
fprintf(output, _("\nFor more information, type \"\\?\" (for internal commands) or \"\\help\" (for SQL\n"
"commands) from within psql, or consult the psql section in the PostgreSQL\n"
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index be57574..9271716 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -75,6 +75,7 @@ struct adhoc_opts
bool no_psqlrc;
bool single_txn;
bool list_dbs;
+ bool compression;
SimpleActionList actions;
};
@@ -237,8 +238,10 @@ main(int argc, char *argv[])
values[5] = pset.progname;
keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
- keywords[7] = NULL;
- values[7] = NULL;
+ keywords[7] = "compression";
+ values[7] = options.compression ? "on" : "off";
+ keywords[8] = NULL;
+ values[8] = NULL;
new_pass = false;
pset.db = PQconnectdbParams(keywords, values, true);
@@ -436,6 +439,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
{"echo-all", no_argument, NULL, 'a'},
{"no-align", no_argument, NULL, 'A'},
{"command", required_argument, NULL, 'c'},
+ {"compression", no_argument, NULL, 'Z'},
{"dbname", required_argument, NULL, 'd'},
{"echo-queries", no_argument, NULL, 'e'},
{"echo-errors", no_argument, NULL, 'b'},
@@ -476,7 +480,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
memset(options, 0, sizeof *options);
- while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01",
+ while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01Z",
long_options, &optindex)) != -1)
{
switch (c)
@@ -540,6 +544,9 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
case 'p':
options->port = pg_strdup(optarg);
break;
+ case 'Z':
+ options->compression = true;
+ break;
case 'P':
{
char *value;
diff --git a/src/bin/scripts/pg_isready.c b/src/bin/scripts/pg_isready.c
index f7ad7b4..5d035cd 100644
--- a/src/bin/scripts/pg_isready.c
+++ b/src/bin/scripts/pg_isready.c
@@ -34,7 +34,7 @@ main(int argc, char **argv)
const char *pghostaddr_str = NULL;
const char *pgport_str = NULL;
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..dbb05f2
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,368 @@
+#include "postgres_fe.h"
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx_error;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ int rc;
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ rc = deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ rc = inflateInit(&zs->rx);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx.msg;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+#else
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ return NULL;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size)
+{
+ return -1;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return NULL;
+}
+
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return 0;
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..dc765af
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,28 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..2dad4fb 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,16 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
@@ -49,7 +59,7 @@ endif
# src/backend/utils/mb
OBJS += encnames.o wchar.o
# src/common
-OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o
+OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS += fe-secure-openssl.o fe-secure-common.o sha2_openssl.o
@@ -106,7 +116,7 @@ backend_src = $(top_srcdir)/src/backend
chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c: % : $(top_srcdir)/src/port/%
rm -f $@ && $(LN_S) $< .
-ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c: % : $(top_srcdir)/src/common/%
+ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c: % : $(top_srcdir)/src/common/%
rm -f $@ && $(LN_S) $< .
encnames.c wchar.c: % : $(backend_src)/utils/mb/%
@@ -156,7 +166,7 @@ clean distclean: clean-lib
rm -f pg_config_paths.h
# Remove files we (may have) symlinked in from src/port and other places
rm -f chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c
- rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c
+ rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c
rm -f encnames.c wchar.c
maintainer-clean: distclean maintainer-clean-lib
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..6fabc5b 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -269,6 +270,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
21, /* sizeof("tls-server-end-point") == 21 */
offsetof(struct pg_conn, scram_channel_binding)},
+ {"compression", "COMPRESSION", "0", NULL,
+ "ZSTD-Compression", "", 1,
+ offsetof(struct pg_conn, compression)},
+
/*
* ssl options are allowed even without client SSL support because the
* client can still handle SSL modes "disable" and "allow". Other
@@ -325,6 +330,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", NULL, NULL, NULL,
+ "Compression", "Z", 5,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +439,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2661,23 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3487,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-18 20:34 ` Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Robbie Harwood @ 2018-06-18 20:34 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
t
Konstantin Knizhnik <[email protected]> writes:
> On 06.06.2018 02:03, Thomas Munro wrote:
>> On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik
>> <[email protected]> wrote:
>>> Thank you for review. Updated version of the patch fixing all reported
>>> problems is attached.
>> Small problem on Windows[1]:
>>
>> C:\projects\postgresql\src\include\common/zpq_stream.h(17): error
>> C2143: syntax error : missing ')' before '*'
>> [C:\projects\postgresql\libpq.vcxproj]
>> 2395
>>
>> You used ssize_t in zpq_stream.h, but Windows doesn't have that type.
>> We have our own typedef in win32_port.h. Perhaps zpq_stream.c should
>> include postgres.h/postgres_fe.h (depending on FRONTEND) like the
>> other .c files in src/common, before it includes zpq_stream.h?
>> Instead of "c.h".
>>
>> [1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.1106
>>
> Thank you very much for reporting the problem.
> I attached new patch with include of postgres_fe.h added to zpq_stream.c
Hello!
Due to being in a similar place, I'm offering some code review. I'm
excited that you're looking at throughput on the network stack - it's
not usually what we think of in database performance. Here are some
first thoughts, which have some overlap with what others have said on
this thread already:
###
This build still doesn't pass Windows:
https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.2277
You can find more about what the bot is doing here:
http://cfbot.cputube.org/
###
I have a few misgivings about pq_configure(), starting with the name.
The *only* thing this function does is set up compression, so it's
mis-named. (There's no point in making something generic unless it's
needed - it's just confusing.)
I also don't like that you've injected into the *startup* path - before
authentication takes place. Fundamentally, authentication (if it
happens) consists of exchanging some combination of short strings (e.g.,
usernames) and binary blobs (e.g., keys). None of this will compress
well, so I don't see it as worth performing this negotiation there - it
can wait. It's also another message in every startup. I'd leave it to
connection parameters, personally, but up to you.
###
Documentation! You're going to need it. There needs to be enough
around for other people to implement the protocol (or if you prefer,
enough for us to debug the protocol as it exists).
In conjunction with that, please add information on how to set up
compressed vs. uncompressed connections - similarly to how we've
documentation on setting up TLS connection (though presumably compressed
connection documentation will be shorter).
###
Using terminology from https://facebook.github.io/zstd/zstd_manual.html :
Right now you use streaming (ZSTD_{compress,decompress}Stream()) as the
basis for your API. I think this is probably a mismatch for what we're
doing here - we're doing one-shot compression/decompression of packets,
not sending video or something.
I think our use case is better served by the non-streaming interface, or
what they call the "Simple API" (ZSTD_{decompress,compress}()).
Documentation suggests it may be worth it to keep an explicit context
around and use that interface instead (i.e.,
ZSTD_{compressCCTx,decompressDCtx}()), but that's something you'll have
to figure out.
You may find making this change helpful for addressing the next issue.
###
I don't like where you've put the entry points to the compression logic:
it's a layering violation. A couple people have expressed similar
reservations I think, so let me see if I can explain using
`pqsecure_read()` as an example. In pseudocode, `pqsecure_read()` looks
like this:
if conn->is_tls:
n = tls_read(conn, ptr, len)
else:
n = pqsecure_raw_read(conn, ptr, len)
return n
I want to see this extended by your code to something like:
if conn->is_tls:
n = tls_read(conn, ptr, len)
else:
n = pqsecure_raw_read(conn, ptr, len)
if conn->is_compressed:
n = decompress(ptr, n)
return n
In conjunction with the above change, this should also significantly
reduce the size of the patch (I think).
###
The compression flag has proven somewhat contentious, as you've already
seen on this thread. I recommend removing it for now and putting it in
a separate patch to be merged later, since it's separable.
###
It's not worth flagging style violations in your code right now, but you
should be aware that there are quite a few style and whitespace
problems. In particular, please be sure that you're using hard tabs
when appropriate, and that line lengths are fewer than 80 characters
(unless they contain error messages), and that pointers are correctly
declared (`void *arg`, not `void* arg`).
###
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-19 07:54 ` Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-19 07:54 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 18.06.2018 23:34, Robbie Harwood wrote:
> t
>
>
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 06.06.2018 02:03, Thomas Munro wrote:
>>> On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik
>>> <[email protected]> wrote:
>>>> Thank you for review. Updated version of the patch fixing all reported
>>>> problems is attached.
>>> Small problem on Windows[1]:
>>>
>>> C:\projects\postgresql\src\include\common/zpq_stream.h(17): error
>>> C2143: syntax error : missing ')' before '*'
>>> [C:\projects\postgresql\libpq.vcxproj]
>>> 2395
>>>
>>> You used ssize_t in zpq_stream.h, but Windows doesn't have that type.
>>> We have our own typedef in win32_port.h. Perhaps zpq_stream.c should
>>> include postgres.h/postgres_fe.h (depending on FRONTEND) like the
>>> other .c files in src/common, before it includes zpq_stream.h?
>>> Instead of "c.h".
>>>
>>> [1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.1106
>>>
>> Thank you very much for reporting the problem.
>> I attached new patch with include of postgres_fe.h added to zpq_stream.c
> Hello!
>
> Due to being in a similar place, I'm offering some code review. I'm
> excited that you're looking at throughput on the network stack - it's
> not usually what we think of in database performance. Here are some
> first thoughts, which have some overlap with what others have said on
> this thread already:
>
> ###
>
> This build still doesn't pass Windows:
> https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.2277
>
> You can find more about what the bot is doing here:
> http://cfbot.cputube.org/
Thank you.
Looks like I found the reason: Mkvsbuild.pm has to be patched.
>
> ###
>
> I have a few misgivings about pq_configure(), starting with the name.
> The *only* thing this function does is set up compression, so it's
> mis-named. (There's no point in making something generic unless it's
> needed - it's just confusing.)
Well, my intention was that this function *may* in future perform some
other configuration setting not related with compression.
And it is better to encapsulate this knowledge in pqcomm, rather than
make postmaster (BackendStartup) worry about it.
But I can rename this function to pq_cofigure_compression() or whatever
your prefer.
> I also don't like that you've injected into the *startup* path - before
> authentication takes place. Fundamentally, authentication (if it
> happens) consists of exchanging some combination of short strings (e.g.,
> usernames) and binary blobs (e.g., keys). None of this will compress
> well, so I don't see it as worth performing this negotiation there - it
> can wait. It's also another message in every startup. I'd leave it to
> connection parameters, personally, but up to you.
From my point of view compression of libpq traffic is similar with SSL
and should be toggled at the same stage.
Definitely authentication parameter are not so large to be efficiently
compressed, by compression (may be in future password protected) can
somehow protect this data.
In any case I do not think that compression of authentication data may
have any influence on negotiation speed.
So I am not 100% sure that toggling compression just after receiving
startup package is the only right solution.
But I am not also convinced in that there is some better place where
compressor should be configured.
Do you have some concrete suggestions for it? In InitPostgres just after
PerformAuthentication ?
Also please notice that compression is useful not only for client-server
communication, but also for replication channels.
Right now it is definitely used in both cases, but if we move
pq_configure somewhere else, we should check that this code is invoked
in both for normal backends and walsender.
>
> ###
>
> Documentation! You're going to need it. There needs to be enough
> around for other people to implement the protocol (or if you prefer,
> enough for us to debug the protocol as it exists).
>
> In conjunction with that, please add information on how to set up
> compressed vs. uncompressed connections - similarly to how we've
> documentation on setting up TLS connection (though presumably compressed
> connection documentation will be shorter).
Sorry, definitely I will add documentation for configuring compression.
>
> ###
>
> Using terminology from https://facebook.github.io/zstd/zstd_manual.html :
>
> Right now you use streaming (ZSTD_{compress,decompress}Stream()) as the
> basis for your API. I think this is probably a mismatch for what we're
> doing here - we're doing one-shot compression/decompression of packets,
> not sending video or something.
>
> I think our use case is better served by the non-streaming interface, or
> what they call the "Simple API" (ZSTD_{decompress,compress}()).
> Documentation suggests it may be worth it to keep an explicit context
> around and use that interface instead (i.e.,
> ZSTD_{compressCCTx,decompressDCtx}()), but that's something you'll have
> to figure out.
>
> You may find making this change helpful for addressing the next issue.
Sorry, but here I completely disagree with you.
What we are doing is really streaming compression, not compression of
individual messages/packages.
Yes, it is not a video, but actually COPY data has the same nature as
video data.
The main benefit of streaming compression is that we can use the same
dictionary for compressing all messages (and adjust this dictionary
based on new data).
We do not need to write dictionary and separate header for each record.
Otherwize compression of libpq messages will be completely useless:
typical size of message is too short to be efficiently compressed. The
main drawback of streaming compression is that you can not decompress
some particular message without decompression of all previous messages.
This is why streaming compression can not be used to compress database
pages (as it is done in CFS, provided in PostgresPro EE). But for libpq
it is no needed.
> ###
>
> I don't like where you've put the entry points to the compression logic:
> it's a layering violation. A couple people have expressed similar
> reservations I think, so let me see if I can explain using
> `pqsecure_read()` as an example. In pseudocode, `pqsecure_read()` looks
> like this:
>
> if conn->is_tls:
> n = tls_read(conn, ptr, len)
> else:
> n = pqsecure_raw_read(conn, ptr, len)
> return n
>
> I want to see this extended by your code to something like:
>
> if conn->is_tls:
> n = tls_read(conn, ptr, len)
> else:
> n = pqsecure_raw_read(conn, ptr, len)
>
> if conn->is_compressed:
> n = decompress(ptr, n)
>
> return n
>
> In conjunction with the above change, this should also significantly
> reduce the size of the patch (I think).
Yes, it will simplify patch. But make libpq compression completely
useless (see my explanation above).
We need to use streaming compression, and to be able to use streaming
compression I have to pass function for fetching more data to
compression library.
>
> ###
>
> The compression flag has proven somewhat contentious, as you've already
> seen on this thread. I recommend removing it for now and putting it in
> a separate patch to be merged later, since it's separable.
>
> ###
>
> It's not worth flagging style violations in your code right now, but you
> should be aware that there are quite a few style and whitespace
> problems. In particular, please be sure that you're using hard tabs
> when appropriate, and that line lengths are fewer than 80 characters
> (unless they contain error messages), and that pointers are correctly
> declared (`void *arg`, not `void* arg`).
>
> ###
Ok, I will fix it.
>
> Thanks,
> --Robbie
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] libpq-compression-6.patch (37.7K, ../../[email protected]/3-libpq-compression-6.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..c963657 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,31 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only conpression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression = 'z'; /* Request compression message */
+ int rc;
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, 1)) < 0 && errno == EINTR);
+ if (rc != 1)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +254,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +312,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +965,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +988,37 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ char const* msg = zpq_error(PqStream);
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1039,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1054,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1073,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1094,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1119,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1153,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1194,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1444,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1503,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f0c5149..efc4ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -188,6 +188,7 @@ int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
+bool libpq_compression; /* use libpq compression */
int main_pid; /* main process id used in log filename */
char *pghost = "";
@@ -590,6 +591,7 @@ usage(void)
" -h, --host=HOSTNAME database server host or socket directory\n"
" -p, --port=PORT database server port number\n"
" -U, --username=USERNAME connect as specified database user\n"
+ " -Z, --compression use libpq compression\n"
" -V, --version output version information, then exit\n"
" -?, --help show this help, then exit\n"
"\n"
@@ -1107,7 +1109,7 @@ doConnect(void)
*/
do
{
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
@@ -1124,8 +1126,10 @@ doConnect(void)
values[4] = dbName;
keywords[5] = "fallback_application_name";
values[5] = progname;
- keywords[6] = NULL;
- values[6] = NULL;
+ keywords[6] = "compression";
+ values[6] = libpq_compression ? "on" : "off";
+ keywords[7] = NULL;
+ values[7] = NULL;
new_pass = false;
@@ -4759,6 +4763,7 @@ main(int argc, char **argv)
{"builtin", required_argument, NULL, 'b'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
+ {"compression", no_argument, NULL, 'Z'},
{"debug", no_argument, NULL, 'd'},
{"define", required_argument, NULL, 'D'},
{"file", required_argument, NULL, 'f'},
@@ -4868,12 +4873,15 @@ main(int argc, char **argv)
exit(1);
}
- while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:Z", long_options, &optindex)) != -1)
{
char *script;
switch (c)
{
+ case 'Z':
+ libpq_compression = true;
+ break;
case 'i':
is_init_mode = true;
break;
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 702e742..ae7a14c 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -139,6 +139,7 @@ usage(unsigned short int pager)
fprintf(output, _(" -U, --username=USERNAME database user name (default: \"%s\")\n"), env);
fprintf(output, _(" -w, --no-password never prompt for password\n"));
fprintf(output, _(" -W, --password force password prompt (should happen automatically)\n"));
+ fprintf(output, _(" -Z, --compression compress traffic with server\n"));
fprintf(output, _("\nFor more information, type \"\\?\" (for internal commands) or \"\\help\" (for SQL\n"
"commands) from within psql, or consult the psql section in the PostgreSQL\n"
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index be57574..9271716 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -75,6 +75,7 @@ struct adhoc_opts
bool no_psqlrc;
bool single_txn;
bool list_dbs;
+ bool compression;
SimpleActionList actions;
};
@@ -237,8 +238,10 @@ main(int argc, char *argv[])
values[5] = pset.progname;
keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
- keywords[7] = NULL;
- values[7] = NULL;
+ keywords[7] = "compression";
+ values[7] = options.compression ? "on" : "off";
+ keywords[8] = NULL;
+ values[8] = NULL;
new_pass = false;
pset.db = PQconnectdbParams(keywords, values, true);
@@ -436,6 +439,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
{"echo-all", no_argument, NULL, 'a'},
{"no-align", no_argument, NULL, 'A'},
{"command", required_argument, NULL, 'c'},
+ {"compression", no_argument, NULL, 'Z'},
{"dbname", required_argument, NULL, 'd'},
{"echo-queries", no_argument, NULL, 'e'},
{"echo-errors", no_argument, NULL, 'b'},
@@ -476,7 +480,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
memset(options, 0, sizeof *options);
- while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01",
+ while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01Z",
long_options, &optindex)) != -1)
{
switch (c)
@@ -540,6 +544,9 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
case 'p':
options->port = pg_strdup(optarg);
break;
+ case 'Z':
+ options->compression = true;
+ break;
case 'P':
{
char *value;
diff --git a/src/bin/scripts/pg_isready.c b/src/bin/scripts/pg_isready.c
index f7ad7b4..5d035cd 100644
--- a/src/bin/scripts/pg_isready.c
+++ b/src/bin/scripts/pg_isready.c
@@ -34,7 +34,7 @@ main(int argc, char **argv)
const char *pghostaddr_str = NULL;
const char *pgport_str = NULL;
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..dbb05f2
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,368 @@
+#include "postgres_fe.h"
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx_error;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ int rc;
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ rc = deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ rc = inflateInit(&zs->rx);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx.msg;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+#else
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ return NULL;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size)
+{
+ return -1;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return NULL;
+}
+
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return 0;
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..dc765af
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,28 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..2dad4fb 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,16 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
@@ -49,7 +59,7 @@ endif
# src/backend/utils/mb
OBJS += encnames.o wchar.o
# src/common
-OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o
+OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS += fe-secure-openssl.o fe-secure-common.o sha2_openssl.o
@@ -106,7 +116,7 @@ backend_src = $(top_srcdir)/src/backend
chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c: % : $(top_srcdir)/src/port/%
rm -f $@ && $(LN_S) $< .
-ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c: % : $(top_srcdir)/src/common/%
+ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c: % : $(top_srcdir)/src/common/%
rm -f $@ && $(LN_S) $< .
encnames.c wchar.c: % : $(backend_src)/utils/mb/%
@@ -156,7 +166,7 @@ clean distclean: clean-lib
rm -f pg_config_paths.h
# Remove files we (may have) symlinked in from src/port and other places
rm -f chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c
- rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c
+ rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c
rm -f encnames.c wchar.c
maintainer-clean: distclean maintainer-clean-lib
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..6fabc5b 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -269,6 +270,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
21, /* sizeof("tls-server-end-point") == 21 */
offsetof(struct pg_conn, scram_channel_binding)},
+ {"compression", "COMPRESSION", "0", NULL,
+ "ZSTD-Compression", "", 1,
+ offsetof(struct pg_conn, compression)},
+
/*
* ssl options are allowed even without client SSL support because the
* client can still handle SSL modes "disable" and "allow". Other
@@ -325,6 +330,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", NULL, NULL, NULL,
+ "Compression", "Z", 5,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +439,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2661,23 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3487,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 593732f..ab2bad5 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -116,7 +116,7 @@ sub mkvcbuild
our @pgcommonallfiles = qw(
base64.c config_info.c controldata_utils.c exec.c file_perm.c ip.c
- keywords.c md5.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
+ keywords.c md5.c zpq_stream.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
saslprep.c scram-common.c string.c unicode_norm.c username.c
wait_error.c);
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-19 21:04 ` Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-19 21:04 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 18.06.2018 23:34, Robbie Harwood wrote:
>
>> I also don't like that you've injected into the *startup* path -
>> before authentication takes place. Fundamentally, authentication (if
>> it happens) consists of exchanging some combination of short strings
>> (e.g., usernames) and binary blobs (e.g., keys). None of this will
>> compress well, so I don't see it as worth performing this negotiation
>> there - it can wait. It's also another message in every startup.
>> I'd leave it to connection parameters, personally, but up to you.
>
> From my point of view compression of libpq traffic is similar with SSL
> and should be toggled at the same stage.
But that's not what you're doing. This isn't where TLS gets toggled.
TLS negotiation happens as the very first packet: after completing the
TCP handshake, the client will send a TLS negotiation request. If it
doesn't happen there, it doesn't happen at all.
(You *could* configure it where TLS is toggled. This is, I think, not a
good idea. TLS encryption is a probe: the server can reject it, at
which point the client tears everything down and connects without TLS.
So if you do the same with compression, that's another point of tearing
down an starting over. The scaling on it isn't good either: if we add
another encryption method into the mix, you've doubled the number of
teardowns.)
> Definitely authentication parameter are not so large to be efficiently
> compressed, by compression (may be in future password protected) can
> somehow protect this data.
> In any case I do not think that compression of authentication data may
> have any influence on negotiation speed.
> So I am not 100% sure that toggling compression just after receiving
> startup package is the only right solution.
> But I am not also convinced in that there is some better place where
> compressor should be configured.
> Do you have some concrete suggestions for it? In InitPostgres just after
> PerformAuthentication ?
Hmm, let me try to explain this differently.
pq_configure() (as you've called it) shouldn't send a packet. At its
callsite, we *already know* whether we want to use compression - that's
what the port->use_compression option says. So there's no point in
having a negotiation there - it's already happened.
The other thing you do in pq_configure() is call zpq_create(), which
does a bunch of initialization for you. I am pretty sure that all of
this can be deferred until the first time you want to send a compressed
message - i.e., when compress()/decompress() is called for the first
time from *secure_read() or *secure_write().
> Also please notice that compression is useful not only for client-server
> communication, but also for replication channels.
> Right now it is definitely used in both cases, but if we move
> pq_configure somewhere else, we should check that this code is invoked
> in both for normal backends and walsender.
"We" meaning you, at the moment, since I don't think any of the rest of
us have set up tests with this code :)
If there's common code to be shared around, that's great. But it's not
imperative; in a lot of ways, the network stacks are very different from
each other, as I'm sure you've seen. Let's not have the desire for code
reuse get in the way of good, maintainable design.
>> Using terminology from https://facebook.github.io/zstd/zstd_manual.html :
>>
>> Right now you use streaming (ZSTD_{compress,decompress}Stream()) as the
>> basis for your API. I think this is probably a mismatch for what we're
>> doing here - we're doing one-shot compression/decompression of packets,
>> not sending video or something.
>>
>> I think our use case is better served by the non-streaming interface, or
>> what they call the "Simple API" (ZSTD_{decompress,compress}()).
>> Documentation suggests it may be worth it to keep an explicit context
>> around and use that interface instead (i.e.,
>> ZSTD_{compressCCTx,decompressDCtx}()), but that's something you'll have
>> to figure out.
>>
>> You may find making this change helpful for addressing the next issue.
>
> Sorry, but here I completely disagree with you.
> What we are doing is really streaming compression, not compression of
> individual messages/packages.
> Yes, it is not a video, but actually COPY data has the same nature as
> video data.
> The main benefit of streaming compression is that we can use the same
> dictionary for compressing all messages (and adjust this dictionary
> based on new data).
> We do not need to write dictionary and separate header for each record.
> Otherwize compression of libpq messages will be completely useless:
> typical size of message is too short to be efficiently compressed. The
> main drawback of streaming compression is that you can not decompress
> some particular message without decompression of all previous messages.
> This is why streaming compression can not be used to compress database
> pages (as it is done in CFS, provided in PostgresPro EE). But for libpq
> it is no needed.
That makes sense, thanks. The zstd documentation doesn't articulate
that at all.
>> I don't like where you've put the entry points to the compression logic:
>> it's a layering violation. A couple people have expressed similar
>> reservations I think, so let me see if I can explain using
>> `pqsecure_read()` as an example. In pseudocode, `pqsecure_read()` looks
>> like this:
>>
>> if conn->is_tls:
>> n = tls_read(conn, ptr, len)
>> else:
>> n = pqsecure_raw_read(conn, ptr, len)
>> return n
>>
>> I want to see this extended by your code to something like:
>>
>> if conn->is_tls:
>> n = tls_read(conn, ptr, len)
>> else:
>> n = pqsecure_raw_read(conn, ptr, len)
>>
>> if conn->is_compressed:
>> n = decompress(ptr, n)
>>
>> return n
>>
>> In conjunction with the above change, this should also significantly
>> reduce the size of the patch (I think).
>
> Yes, it will simplify patch. But make libpq compression completely
> useless (see my explanation above).
> We need to use streaming compression, and to be able to use streaming
> compression I have to pass function for fetching more data to
> compression library.
I don't think you need that, even with the streaming API.
To make this very concrete, let's talk about ZSTD_decompressStream (I'm
pulling information from
https://facebook.github.io/zstd/zstd_manual.html#Chapter7 ).
Using the pseudocode I'm asking for above, the decompress() function
would look vaguely like this:
decompress(ptr, n)
ZSTD_inBuffer in = {0}
ZSTD_outBuffer out = {0}
in.src = ptr
in.size = n
while in.pos < in.size:
ret = ZSTD_decompressStream(out, in)
if ZSTD_isError(ret):
give_up()
memcpy(ptr, out.dst, out.pos)
return out.pos
(and compress() would follow a similar pattern, if we were to talk about
it).
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-20 06:40 ` Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-20 06:40 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 20.06.2018 00:04, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 18.06.2018 23:34, Robbie Harwood wrote:
>>
>>> I also don't like that you've injected into the *startup* path -
>>> before authentication takes place. Fundamentally, authentication (if
>>> it happens) consists of exchanging some combination of short strings
>>> (e.g., usernames) and binary blobs (e.g., keys). None of this will
>>> compress well, so I don't see it as worth performing this negotiation
>>> there - it can wait. It's also another message in every startup.
>>> I'd leave it to connection parameters, personally, but up to you.
>> From my point of view compression of libpq traffic is similar with SSL
>> and should be toggled at the same stage.
> But that's not what you're doing. This isn't where TLS gets toggled.
>
> TLS negotiation happens as the very first packet: after completing the
> TCP handshake, the client will send a TLS negotiation request. If it
> doesn't happen there, it doesn't happen at all.
>
> (You *could* configure it where TLS is toggled. This is, I think, not a
> good idea. TLS encryption is a probe: the server can reject it, at
> which point the client tears everything down and connects without TLS.
> So if you do the same with compression, that's another point of tearing
> down an starting over. The scaling on it isn't good either: if we add
> another encryption method into the mix, you've doubled the number of
> teardowns.)
Yes, you are right. There is special message for enabling TLS procotol.
But I do not think that the same think is needed for compression.
This is why I prefer to specify compression in connectoin options. So
compression may be enabled straight after processing of startup package.
Frankly speaking I still do no see reasons to postpone enabling
compression till some later moment.
>> Definitely authentication parameter are not so large to be efficiently
>> compressed, by compression (may be in future password protected) can
>> somehow protect this data.
>> In any case I do not think that compression of authentication data may
>> have any influence on negotiation speed.
>> So I am not 100% sure that toggling compression just after receiving
>> startup package is the only right solution.
>> But I am not also convinced in that there is some better place where
>> compressor should be configured.
>> Do you have some concrete suggestions for it? In InitPostgres just after
>> PerformAuthentication ?
> Hmm, let me try to explain this differently.
>
> pq_configure() (as you've called it) shouldn't send a packet. At its
> callsite, we *already know* whether we want to use compression - that's
> what the port->use_compression option says. So there's no point in
> having a negotiation there - it's already happened.
My idea was the following: client want to use compression. But server
may reject this attempt (for any reasons: it doesn't support it, has no
proper compression library,
do not want to spend CPU for decompression,...) Right now compression
algorithm is hardcoded. But in future client and server may negotiate to
choose proper compression protocol.
This is why I prefer to perform negotiation between client and server to
enable compression.
>
> The other thing you do in pq_configure() is call zpq_create(), which
> does a bunch of initialization for you. I am pretty sure that all of
> this can be deferred until the first time you want to send a compressed
> message - i.e., when compress()/decompress() is called for the first
> time from *secure_read() or *secure_write().
>
>> Also please notice that compression is useful not only for client-server
>> communication, but also for replication channels.
>> Right now it is definitely used in both cases, but if we move
>> pq_configure somewhere else, we should check that this code is invoked
>> in both for normal backends and walsender.
> "We" meaning you, at the moment, since I don't think any of the rest of
> us have set up tests with this code :)
>
> If there's common code to be shared around, that's great. But it's not
> imperative; in a lot of ways, the network stacks are very different from
> each other, as I'm sure you've seen. Let's not have the desire for code
> reuse get in the way of good, maintainable design.
>
>>> Using terminology from https://facebook.github.io/zstd/zstd_manual.html :
>>>
>>> Right now you use streaming (ZSTD_{compress,decompress}Stream()) as the
>>> basis for your API. I think this is probably a mismatch for what we're
>>> doing here - we're doing one-shot compression/decompression of packets,
>>> not sending video or something.
>>>
>>> I think our use case is better served by the non-streaming interface, or
>>> what they call the "Simple API" (ZSTD_{decompress,compress}()).
>>> Documentation suggests it may be worth it to keep an explicit context
>>> around and use that interface instead (i.e.,
>>> ZSTD_{compressCCTx,decompressDCtx}()), but that's something you'll have
>>> to figure out.
>>>
>>> You may find making this change helpful for addressing the next issue.
>> Sorry, but here I completely disagree with you.
>> What we are doing is really streaming compression, not compression of
>> individual messages/packages.
>> Yes, it is not a video, but actually COPY data has the same nature as
>> video data.
>> The main benefit of streaming compression is that we can use the same
>> dictionary for compressing all messages (and adjust this dictionary
>> based on new data).
>> We do not need to write dictionary and separate header for each record.
>> Otherwize compression of libpq messages will be completely useless:
>> typical size of message is too short to be efficiently compressed. The
>> main drawback of streaming compression is that you can not decompress
>> some particular message without decompression of all previous messages.
>> This is why streaming compression can not be used to compress database
>> pages (as it is done in CFS, provided in PostgresPro EE). But for libpq
>> it is no needed.
> That makes sense, thanks. The zstd documentation doesn't articulate
> that at all.
>
>>> I don't like where you've put the entry points to the compression logic:
>>> it's a layering violation. A couple people have expressed similar
>>> reservations I think, so let me see if I can explain using
>>> `pqsecure_read()` as an example. In pseudocode, `pqsecure_read()` looks
>>> like this:
>>>
>>> if conn->is_tls:
>>> n = tls_read(conn, ptr, len)
>>> else:
>>> n = pqsecure_raw_read(conn, ptr, len)
>>> return n
>>>
>>> I want to see this extended by your code to something like:
>>>
>>> if conn->is_tls:
>>> n = tls_read(conn, ptr, len)
>>> else:
>>> n = pqsecure_raw_read(conn, ptr, len)
>>>
>>> if conn->is_compressed:
>>> n = decompress(ptr, n)
>>>
>>> return n
>>>
>>> In conjunction with the above change, this should also significantly
>>> reduce the size of the patch (I think).
>> Yes, it will simplify patch. But make libpq compression completely
>> useless (see my explanation above).
>> We need to use streaming compression, and to be able to use streaming
>> compression I have to pass function for fetching more data to
>> compression library.
> I don't think you need that, even with the streaming API.
>
> To make this very concrete, let's talk about ZSTD_decompressStream (I'm
> pulling information from
> https://facebook.github.io/zstd/zstd_manual.html#Chapter7 ).
>
> Using the pseudocode I'm asking for above, the decompress() function
> would look vaguely like this:
>
> decompress(ptr, n)
> ZSTD_inBuffer in = {0}
> ZSTD_outBuffer out = {0}
>
> in.src = ptr
> in.size = n
>
> while in.pos < in.size:
> ret = ZSTD_decompressStream(out, in)
> if ZSTD_isError(ret):
> give_up()
>
> memcpy(ptr, out.dst, out.pos)
> return out.pos
>
> (and compress() would follow a similar pattern, if we were to talk about
> it).
It will not work in this way. We do not know how much input data we need
to be able to decompress message.
So loop should be something like this:
decompress(ptr, n)
ZSTD_inBuffer in = {0}
ZSTD_outBuffer out = {0}
in.src = ptr
in.size = n
while true
ret = ZSTD_decompressStream(out, in)
if ZSTD_isError(ret):
give_up()
if out.pos != 0
// if we deomcpress soemthing
return out.pos;
read_input(in);
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-20 20:34 ` Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-20 20:34 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 20.06.2018 00:04, Robbie Harwood wrote:
>> Konstantin Knizhnik <[email protected]> writes:
>>> On 18.06.2018 23:34, Robbie Harwood wrote:
>>>
>>>> I also don't like that you've injected into the *startup* path -
>>>> before authentication takes place. Fundamentally, authentication
>>>> (if it happens) consists of exchanging some combination of short
>>>> strings (e.g., usernames) and binary blobs (e.g., keys). None of
>>>> this will compress well, so I don't see it as worth performing this
>>>> negotiation there - it can wait. It's also another message in
>>>> every startup. I'd leave it to connection parameters, personally,
>>>> but up to you.
>>>
>>> From my point of view compression of libpq traffic is similar with
>>> SSL and should be toggled at the same stage.
>>
>> But that's not what you're doing. This isn't where TLS gets toggled.
>>
>> TLS negotiation happens as the very first packet: after completing
>> the TCP handshake, the client will send a TLS negotiation request.
>> If it doesn't happen there, it doesn't happen at all.
>>
>> (You *could* configure it where TLS is toggled. This is, I think,
>> not a good idea. TLS encryption is a probe: the server can reject
>> it, at which point the client tears everything down and connects
>> without TLS. So if you do the same with compression, that's another
>> point of tearing down an starting over. The scaling on it isn't good
>> either: if we add another encryption method into the mix, you've
>> doubled the number of teardowns.)
>
> Yes, you are right. There is special message for enabling TLS
> procotol. But I do not think that the same think is needed for
> compression. This is why I prefer to specify compression in
> connectoin options. So compression may be enabled straight after
> processing of startup package. Frankly speaking I still do no see
> reasons to postpone enabling compression till some later moment.
I'm arguing for connection option only (with no additional negotiation
round-trip). See below.
>>> Definitely authentication parameter are not so large to be
>>> efficiently compressed, by compression (may be in future password
>>> protected) can somehow protect this data. In any case I do not
>>> think that compression of authentication data may have any influence
>>> on negotiation speed. So I am not 100% sure that toggling
>>> compression just after receiving startup package is the only right
>>> solution. But I am not also convinced in that there is some better
>>> place where compressor should be configured. Do you have some
>>> concrete suggestions for it? In InitPostgres just after
>>> PerformAuthentication ?
>>
>> Hmm, let me try to explain this differently.
>>
>> pq_configure() (as you've called it) shouldn't send a packet. At its
>> callsite, we *already know* whether we want to use compression -
>> that's what the port->use_compression option says. So there's no
>> point in having a negotiation there - it's already happened.
>
> My idea was the following: client want to use compression. But server
> may reject this attempt (for any reasons: it doesn't support it, has
> no proper compression library, do not want to spend CPU for
> decompression,...) Right now compression algorithm is hardcoded. But
> in future client and server may negotiate to choose proper compression
> protocol. This is why I prefer to perform negotiation between client
> and server to enable compression.
Well, for negotiation you could put the name of the algorithm you want
in the startup. It doesn't have to be a boolean for compression, and
then you don't need an additional round-trip.
>>>> I don't like where you've put the entry points to the compression
>>>> logic: it's a layering violation. A couple people have expressed
>>>> similar reservations I think, so let me see if I can explain using
>>>> `pqsecure_read()` as an example. In pseudocode, `pqsecure_read()`
>>>> looks like this:
>>>>
>>>> if conn->is_tls:
>>>> n = tls_read(conn, ptr, len)
>>>> else:
>>>> n = pqsecure_raw_read(conn, ptr, len)
>>>> return n
>>>>
>>>> I want to see this extended by your code to something like:
>>>>
>>>> if conn->is_tls:
>>>> n = tls_read(conn, ptr, len)
>>>> else:
>>>> n = pqsecure_raw_read(conn, ptr, len)
>>>>
>>>> if conn->is_compressed:
>>>> n = decompress(ptr, n)
>>>>
>>>> return n
>>>>
>>>> In conjunction with the above change, this should also
>>>> significantly reduce the size of the patch (I think).
>>>
>>> Yes, it will simplify patch. But make libpq compression completely
>>> useless (see my explanation above). We need to use streaming
>>> compression, and to be able to use streaming compression I have to
>>> pass function for fetching more data to compression library.
>>
>> I don't think you need that, even with the streaming API.
>>
>> To make this very concrete, let's talk about ZSTD_decompressStream (I'm
>> pulling information from
>> https://facebook.github.io/zstd/zstd_manual.html#Chapter7 ).
>>
>> Using the pseudocode I'm asking for above, the decompress() function
>> would look vaguely like this:
>>
>> decompress(ptr, n)
>> ZSTD_inBuffer in = {0}
>> ZpSTD_outBuffer out = {0}
>>
>> in.src = ptr
>> in.size = n
>>
>> while in.pos < in.size:
>> ret = ZSTD_decompressStream(out, in)
>> if ZSTD_isError(ret):
>> give_up()
>>
>> memcpy(ptr, out.dst, out.pos)
>> return out.pos
>>
>> (and compress() would follow a similar pattern, if we were to talk
>> about it).
>
> It will not work in this way. We do not know how much input data we
> need to be able to decompress message.
Well, that's a design decision you've made. You could put lengths on
chunks that are sent out - then you'd know exactly how much is needed.
(For instance, 4 bytes of network-order length followed by a complete
payload.) Then you'd absolutely know whether you have enough to
decompress or not.
> So loop should be something like this:
>
> decompress(ptr, n)
> ZSTD_inBuffer in = {0}
> ZSTD_outBuffer out = {0}
>
> in.src = ptr
> in.size = n
>
> while true
> ret = ZSTD_decompressStream(out, in)
> if ZSTD_isError(ret):
> give_up()
> if out.pos != 0
> // if we deomcpress soemthing
> return out.pos;
> read_input(in);
The last line is what I'm taking issue with. The interface we have
already in postgres's network code has a notion of partial reads, or
that reads might not return data. (Same with writing and partial
writes.) So you'd buffer what compressed data you have and return -
this is the preferred idiom so that we don't block or spin on a
nonblocking socket.
This is how the TLS code works already. Look at, for instance,
pgtls_read(). If we get back SSL_ERROR_WANT_READ (i.e., not enough data
to decrypt), we return no data and wait until the socket becomes
readable again.
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-21 07:12 ` Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 21:34 ` Re: libpq compression Nico Williams <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-21 07:12 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 20.06.2018 23:34, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>
> My idea was the following: client want to use compression. But server
> may reject this attempt (for any reasons: it doesn't support it, has
> no proper compression library, do not want to spend CPU for
> decompression,...) Right now compression algorithm is hardcoded. But
> in future client and server may negotiate to choose proper compression
> protocol. This is why I prefer to perform negotiation between client
> and server to enable compression.
> Well, for negotiation you could put the name of the algorithm you want
> in the startup. It doesn't have to be a boolean for compression, and
> then you don't need an additional round-trip.
Sorry, I can only repeat arguments I already mentioned:
- in future it may be possible to specify compression algorithm
- even with boolean compression option server may have some reasons to
reject client's request to use compression
Extra flexibility is always good thing if it doesn't cost too much. And
extra round of negotiation in case of enabling compression seems to me
not to be a high price for it.
>
> Well, that's a design decision you've made. You could put lengths on
> chunks that are sent out - then you'd know exactly how much is needed.
> (For instance, 4 bytes of network-order length followed by a complete
> payload.) Then you'd absolutely know whether you have enough to
> decompress or not.
Do you really suggest to send extra header for each chunk of data?
Please notice that chunk can be as small as one message: dozen of bytes
because libpq is used for client-server communication with request-reply
pattern.
Frankly speaking I do not completely understand the source of your concern.
My primary idea was to preseve behavior of libpq function as much as
possible, so there is no need to rewrite all places in Postgres code
when them are used.
It seems to me that I succeed in reaching this goal. Incase of enabled
compression zpq_stream functions (zpq-read/write) are used instead of
(pq)secure_read/write and in turn are using them to fetch/send more
data. I do not see any bad flaws, encapsulation violation or some other
problems in such solution.
So before discussing some alternative ways of embedding compression in
libpq, I will want to understand what's wrong with this approach.
>> So loop should be something like this:
>>
>> decompress(ptr, n)
>> ZSTD_inBuffer in = {0}
>> ZSTD_outBuffer out = {0}
>>
>> in.src = ptr
>> in.size = n
>>
>> while true
>> ret = ZSTD_decompressStream(out, in)
>> if ZSTD_isError(ret):
>> give_up()
>> if out.pos != 0
>> // if we deomcpress soemthing
>> return out.pos;
>> read_input(in);
> The last line is what I'm taking issue with. The interface we have
> already in postgres's network code has a notion of partial reads, or
> that reads might not return data. (Same with writing and partial
> writes.) So you'd buffer what compressed data you have and return -
> this is the preferred idiom so that we don't block or spin on a
> nonblocking socket.
If socket is in non-blocking mode and there is available data, then
secure_read function will also immediately return 0.
The pseudocode above is not quite correct. Let me show the real
implementation of zpq_read:
ssize_t
zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
{
   ssize_t rc;
   ZSTD_outBuffer out;
   out.dst = buf;
   out.pos = 0;
   out.size = size;
   while (1)
   {
      rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
      if (ZSTD_isError(rc))
      {
         zs->rx_error = ZSTD_getErrorName(rc);
         return ZPQ_DECOMPRESS_ERROR;
      }
      /* Return result if we fill requested amount of bytes or read
operation was performed */
      if (out.pos != 0)
      {
         zs->rx_total_raw += out.pos;
         return out.pos;
      }
      if (zs->rx.pos == zs->rx.size)
      {
         zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
      }
      rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size,
ZPQ_BUFFER_SIZE - zs->rx.size);
      if (rc > 0) /* read fetches some data */
      {
         zs->rx.size += rc;
         zs->rx_total += rc;
      }
      else /* read failed */
      {
         *processed = out.pos;
         zs->rx_total_raw += out.pos;
         return rc;
      }
   }
}
Sorry, but I have spent quite enough time trying to provide the same
behavior of zpq_read/write as of secure_read/write both for blocking and
non-blocking mode.
And I hope that now it is preserved. And frankly speaking I do not see
much differences of this approach with supporting TLS.
Current implementation allows to combine compression with TLS and in
some cases it may be really useful.
>
> This is how the TLS code works already. Look at, for instance,
> pgtls_read(). If we get back SSL_ERROR_WANT_READ (i.e., not enough data
> to decrypt), we return no data and wait until the socket becomes
> readable again.
>
> Thanks,
> --Robbie
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-21 14:56 ` Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-21 14:56 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 20.06.2018 23:34, Robbie Harwood wrote:
>> Konstantin Knizhnik <[email protected]> writes:
>>
>>
>> My idea was the following: client want to use compression. But server
>> may reject this attempt (for any reasons: it doesn't support it, has
>> no proper compression library, do not want to spend CPU for
>> decompression,...) Right now compression algorithm is hardcoded. But
>> in future client and server may negotiate to choose proper compression
>> protocol. This is why I prefer to perform negotiation between client
>> and server to enable compression.
>> Well, for negotiation you could put the name of the algorithm you want
>> in the startup. It doesn't have to be a boolean for compression, and
>> then you don't need an additional round-trip.
>
> Sorry, I can only repeat arguments I already mentioned:
> - in future it may be possible to specify compression algorithm
> - even with boolean compression option server may have some reasons to
> reject client's request to use compression
>
> Extra flexibility is always good thing if it doesn't cost too much. And
> extra round of negotiation in case of enabling compression seems to me
> not to be a high price for it.
You already have this flexibility even without negotiation. I don't
want you to lose your flexibility. Protocol looks like this:
- Client sends connection option "compression" with list of algorithms
it wants to use (comma-separated, or something).
- First packet that the server can compress one of those algorithms (or
none, if it doesn't want to turn on compression).
No additional round-trips needed.
>> Well, that's a design decision you've made. You could put lengths on
>> chunks that are sent out - then you'd know exactly how much is needed.
>> (For instance, 4 bytes of network-order length followed by a complete
>> payload.) Then you'd absolutely know whether you have enough to
>> decompress or not.
>
> Do you really suggest to send extra header for each chunk of data?
> Please notice that chunk can be as small as one message: dozen of bytes
> because libpq is used for client-server communication with request-reply
> pattern.
I want you to think critically about your design. I *really* don't want
to design it for you - I have enough stuff to be doing. But again, the
design I gave you doesn't necessarily need that - you just need to
properly buffer incomplete data.
> Frankly speaking I do not completely understand the source of your
> concern. My primary idea was to preseve behavior of libpq function as
> much as possible, so there is no need to rewrite all places in
> Postgres code when them are used. It seems to me that I succeed in
> reaching this goal. Incase of enabled compression zpq_stream functions
> (zpq-read/write) are used instead of (pq)secure_read/write and in turn
> are using them to fetch/send more data. I do not see any bad flaws,
> encapsulation violation or some other problems in such solution.
>
> So before discussing some alternative ways of embedding compression in
> libpq, I will want to understand what's wrong with this approach.
You're destroying the existing model for no reason. If you needed to, I
could understand some argument for the way you've done it, but what I've
tried to tell you is that you don't need to do so. It's longer this
way, and it *significantly* complicates the (already difficult to reason
about) connection state machine.
I get that rewriting code can be obnoxious, and it feels like a waste of
time when we have to do so. (I've been there; I'm on version 19 of my
postgres patchset.)
>>> So loop should be something like this:
>>>
>>> decompress(ptr, n)
>>> ZSTD_inBuffer in = {0}
>>> ZSTD_outBuffer out = {0}
>>>
>>> in.src = ptr
>>> in.size = n
>>>
>>> while true
>>> ret = ZSTD_decompressStream(out, in)
>>> if ZSTD_isError(ret):
>>> give_up()
>>> if out.pos != 0
>>> // if we deomcpress soemthing
>>> return out.pos;
>>> read_input(in);
>>
>> The last line is what I'm taking issue with. The interface we have
>> already in postgres's network code has a notion of partial reads, or
>> that reads might not return data. (Same with writing and partial
>> writes.) So you'd buffer what compressed data you have and return -
>> this is the preferred idiom so that we don't block or spin on a
>> nonblocking socket.
>
> If socket is in non-blocking mode and there is available data, then
> secure_read function will also immediately return 0.
> The pseudocode above is not quite correct. Let me show the real
> implementation of zpq_read:
>
> ssize_t
> zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
> {
> Â Â Â ssize_t rc;
> Â Â Â ZSTD_outBuffer out;
> Â Â Â out.dst = buf;
> Â Â Â out.pos = 0;
> Â Â Â out.size = size;
>
> Â Â Â while (1)
> Â Â Â {
> Â Â Â Â Â Â rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
> Â Â Â Â Â Â if (ZSTD_isError(rc))
> Â Â Â Â Â Â {
> Â Â Â Â Â Â Â Â Â zs->rx_error = ZSTD_getErrorName(rc);
> Â Â Â Â Â Â Â Â Â return ZPQ_DECOMPRESS_ERROR;
> Â Â Â Â Â Â }
> Â Â Â Â Â Â /* Return result if we fill requested amount of bytes or read
> operation was performed */
> Â Â Â Â Â Â if (out.pos != 0)
> Â Â Â Â Â Â {
> Â Â Â Â Â Â Â Â Â zs->rx_total_raw += out.pos;
> Â Â Â Â Â Â Â Â Â return out.pos;
> Â Â Â Â Â Â }
> Â Â Â Â Â Â if (zs->rx.pos == zs->rx.size)
> Â Â Â Â Â Â {
> Â Â Â Â Â Â Â Â Â zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
> Â Â Â Â Â Â }
> Â Â Â Â Â Â rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size,
> ZPQ_BUFFER_SIZE - zs->rx.size);
> Â Â Â Â Â Â if (rc > 0) /* read fetches some data */
> Â Â Â Â Â Â {
> Â Â Â Â Â Â Â Â Â zs->rx.size += rc;
> Â Â Â Â Â Â Â Â Â zs->rx_total += rc;
> Â Â Â Â Â Â }
> Â Â Â Â Â Â else /* read failed */
> Â Â Â Â Â Â {
> Â Â Â Â Â Â Â Â Â *processed = out.pos;
> Â Â Â Â Â Â Â Â Â zs->rx_total_raw += out.pos;
> Â Â Â Â Â Â Â Â Â return rc;
> Â Â Â Â Â Â }
> Â Â Â }
> }
>
> Sorry, but I have spent quite enough time trying to provide the same
> behavior of zpq_read/write as of secure_read/write both for blocking and
> non-blocking mode.
> And I hope that now it is preserved. And frankly speaking I do not see
> much differences of this approach with supporting TLS.
>
> Current implementation allows to combine compression with TLS and in
> some cases it may be really useful.
The bottom line, though, is that I cannot recommend the code for
committer as long you have it plumbed like this. -1.
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-21 16:04 ` Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-21 16:04 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 21.06.2018 17:56, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>> Konstantin Knizhnik <[email protected]> writes:
>>>
>>>
>>> My idea was the following: client want to use compression. But server
>>> may reject this attempt (for any reasons: it doesn't support it, has
>>> no proper compression library, do not want to spend CPU for
>>> decompression,...) Right now compression algorithm is hardcoded. But
>>> in future client and server may negotiate to choose proper compression
>>> protocol. This is why I prefer to perform negotiation between client
>>> and server to enable compression.
>>> Well, for negotiation you could put the name of the algorithm you want
>>> in the startup. It doesn't have to be a boolean for compression, and
>>> then you don't need an additional round-trip.
>> Sorry, I can only repeat arguments I already mentioned:
>> - in future it may be possible to specify compression algorithm
>> - even with boolean compression option server may have some reasons to
>> reject client's request to use compression
>>
>> Extra flexibility is always good thing if it doesn't cost too much. And
>> extra round of negotiation in case of enabling compression seems to me
>> not to be a high price for it.
> You already have this flexibility even without negotiation. I don't
> want you to lose your flexibility. Protocol looks like this:
>
> - Client sends connection option "compression" with list of algorithms
> it wants to use (comma-separated, or something).
>
> - First packet that the server can compress one of those algorithms (or
> none, if it doesn't want to turn on compression).
>
> No additional round-trips needed.
This is exactly how it works now...
Client includes compression option in connection string and server
replies with special message ('Z') if it accepts request to compress
traffic between this client and server.
I do not know whether sending such message can be considered as
"round-trip" or not, but I do not want to loose possibility to make
decision at server whether to use compression or not.
>
>>> Well, that's a design decision you've made. You could put lengths on
>>> chunks that are sent out - then you'd know exactly how much is needed.
>>> (For instance, 4 bytes of network-order length followed by a complete
>>> payload.) Then you'd absolutely know whether you have enough to
>>> decompress or not.
>> Do you really suggest to send extra header for each chunk of data?
>> Please notice that chunk can be as small as one message: dozen of bytes
>> because libpq is used for client-server communication with request-reply
>> pattern.
> I want you to think critically about your design. I *really* don't want
> to design it for you - I have enough stuff to be doing. But again, the
> design I gave you doesn't necessarily need that - you just need to
> properly buffer incomplete data.
Right now secure_read may return any number of available bytes. But in
case of using streaming compression, it can happen that available number
of bytes is not enough to perform decompression. This is why we may need
to try to fetch additional portion of data. This is how zpq_stream is
working now.
I do not understand how it is possible to implement in different way and
what is wrong with current implementation.
>
>> Frankly speaking I do not completely understand the source of your
>> concern. My primary idea was to preseve behavior of libpq function as
>> much as possible, so there is no need to rewrite all places in
>> Postgres code when them are used. It seems to me that I succeed in
>> reaching this goal. Incase of enabled compression zpq_stream functions
>> (zpq-read/write) are used instead of (pq)secure_read/write and in turn
>> are using them to fetch/send more data. I do not see any bad flaws,
>> encapsulation violation or some other problems in such solution.
>>
>> So before discussing some alternative ways of embedding compression in
>> libpq, I will want to understand what's wrong with this approach.
> You're destroying the existing model for no reason.
Why? Sorry, I really do not understand why adding compression in this
way breaks exited model.
Can you please explain it to me once again.
> If you needed to, I
> could understand some argument for the way you've done it, but what I've
> tried to tell you is that you don't need to do so. It's longer this
> way, and it *significantly* complicates the (already difficult to reason
> about) connection state machine.
>
> I get that rewriting code can be obnoxious, and it feels like a waste of
> time when we have to do so. (I've been there; I'm on version 19 of my
> postgres patchset.)
I am not against rewriting code many times, but first I should
understand the problem which needs to be solved.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-21 17:14 ` Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-21 17:14 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 21.06.2018 17:56, Robbie Harwood wrote:
>> Konstantin Knizhnik <[email protected]> writes:
>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>
>>>> Well, that's a design decision you've made. You could put lengths
>>>> on chunks that are sent out - then you'd know exactly how much is
>>>> needed. (For instance, 4 bytes of network-order length followed by
>>>> a complete payload.) Then you'd absolutely know whether you have
>>>> enough to decompress or not.
>>>
>>> Do you really suggest to send extra header for each chunk of data?
>>> Please notice that chunk can be as small as one message: dozen of
>>> bytes because libpq is used for client-server communication with
>>> request-reply pattern.
>>
>> I want you to think critically about your design. I *really* don't
>> want to design it for you - I have enough stuff to be doing. But
>> again, the design I gave you doesn't necessarily need that - you just
>> need to properly buffer incomplete data.
>
> Right now secure_read may return any number of available bytes. But in
> case of using streaming compression, it can happen that available
> number of bytes is not enough to perform decompression. This is why we
> may need to try to fetch additional portion of data. This is how
> zpq_stream is working now.
No, you need to buffer and wait until you're called again. Which is to
say: decompress() shouldn't call secure_read(). secure_read() should
call decompress().
> I do not understand how it is possible to implement in different way
> and what is wrong with current implementation.
The most succinct thing I can say is: absolutely don't modify
pq_recvbuf(). I gave you pseudocode for how to do that. All of your
logic should be *below* the secure_read/secure_write functions.
I cannot approve code that modifies pq_recvbuf() in the manner you
currently do.
>>>> My idea was the following: client want to use compression. But
>>>> server may reject this attempt (for any reasons: it doesn't support
>>>> it, has no proper compression library, do not want to spend CPU for
>>>> decompression,...) Right now compression algorithm is
>>>> hardcoded. But in future client and server may negotiate to choose
>>>> proper compression protocol. This is why I prefer to perform
>>>> negotiation between client and server to enable compression. Well,
>>>> for negotiation you could put the name of the algorithm you want in
>>>> the startup. It doesn't have to be a boolean for compression, and
>>>> then you don't need an additional round-trip.
>>>
>>> Sorry, I can only repeat arguments I already mentioned:
>>> - in future it may be possible to specify compression algorithm
>>> - even with boolean compression option server may have some reasons to
>>> reject client's request to use compression
>>>
>>> Extra flexibility is always good thing if it doesn't cost too
>>> much. And extra round of negotiation in case of enabling compression
>>> seems to me not to be a high price for it.
>>
>> You already have this flexibility even without negotiation. I don't
>> want you to lose your flexibility. Protocol looks like this:
>>
>> - Client sends connection option "compression" with list of
>> algorithms it wants to use (comma-separated, or something).
>>
>> - First packet that the server can compress one of those algorithms
>> (or none, if it doesn't want to turn on compression).
>>
>> No additional round-trips needed.
>
> This is exactly how it works now... Client includes compression
> option in connection string and server replies with special message
> ('Z') if it accepts request to compress traffic between this client
> and server.
No, it's not. You don't need this message. If the server receives a
compression request, it should just turn compression on (or not), and
then have the client figure out whether it got compression back. This
is of course made harder by your refusal to use packet framing, but
still shouldn't be particularly difficult.
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-22 07:11 ` Konstantin Knizhnik <[email protected]>
2018-06-22 15:59 ` Re: libpq compression Robbie Harwood <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-22 07:11 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 21.06.2018 20:14, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>> Konstantin Knizhnik <[email protected]> writes:
>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>
>>>>> Well, that's a design decision you've made. You could put lengths
>>>>> on chunks that are sent out - then you'd know exactly how much is
>>>>> needed. (For instance, 4 bytes of network-order length followed by
>>>>> a complete payload.) Then you'd absolutely know whether you have
>>>>> enough to decompress or not.
>>>> Do you really suggest to send extra header for each chunk of data?
>>>> Please notice that chunk can be as small as one message: dozen of
>>>> bytes because libpq is used for client-server communication with
>>>> request-reply pattern.
>>> I want you to think critically about your design. I *really* don't
>>> want to design it for you - I have enough stuff to be doing. But
>>> again, the design I gave you doesn't necessarily need that - you just
>>> need to properly buffer incomplete data.
>> Right now secure_read may return any number of available bytes. But in
>> case of using streaming compression, it can happen that available
>> number of bytes is not enough to perform decompression. This is why we
>> may need to try to fetch additional portion of data. This is how
>> zpq_stream is working now.
> No, you need to buffer and wait until you're called again. Which is to
> say: decompress() shouldn't call secure_read(). secure_read() should
> call decompress().
>
I this case I will have to implement this code twice: both for backend
and frontend, i.e. for secure_read/secure_write and
pqsecure_read/pqsecure_write.
Frankly speaking i was very upset by design of libpq communication layer
in Postgres: there are two different implementations of almost the same
stuff for cbackend and frontend.
I do not see any meaningful argument for it except "historical reasons".
The better decision was to encapsulate socket communication layer (and
some other system dependent stuff) in SAL (system abstraction layer) and
use it both in backend and frontend.
By passing secure_read/pqsecure_read functions to zpq_stream I managed
to avoid such code duplication at least for compression.
>> I do not understand how it is possible to implement in different way
>> and what is wrong with current implementation.
> The most succinct thing I can say is: absolutely don't modify
> pq_recvbuf(). I gave you pseudocode for how to do that. All of your
> logic should be *below* the secure_read/secure_write functions.
>
> I cannot approve code that modifies pq_recvbuf() in the manner you
> currently do.
Well. I understand you arguments.
But please also consider my argument above (about avoiding code
duplication).
In any case, secure_read function is called only from pq_recvbuf() as
well as pqsecure_read is called only from pqReadData.
So I do not think principle difference in handling compression in
secure_read or pq_recvbuf functions and do not understand why it is
"destroying the existing model".
Frankly speaking, I will really like to destroy existed model, moving
all system dependent stuff in Postgres to SAL and avoid this awful mix
of code
sharing and duplication between backend and frontend. But it is a
another story and I do not want to discuss it here.
If we are speaking about the "right design", then neither your
suggestion, neither my implementation are good and I do not see
principle differences between them.
The right approach is using "decorator pattern": this is how streams are
designed in .Net/Java. You can construct pipe of "secure", "compressed"
and whatever else streams.
Yes, it is first of all pattern for object-oriented approach and
Postgres is implemented in C. But it is actually possible to use OO
approach in pure C (X-Windows!).
But once again, this discussion may lead other too far away from the
topic of libpq compression.
As far as I already wrote, the main points of my design were:
1. Minimize changes in Postgres code
2. Avoid code duplication
3. Provide abstract (compression stream) which can be used somewhere
else except libpq itself.
>
>>>>> My idea was the following: client want to use compression. But
>>>>> server may reject this attempt (for any reasons: it doesn't support
>>>>> it, has no proper compression library, do not want to spend CPU for
>>>>> decompression,...) Right now compression algorithm is
>>>>> hardcoded. But in future client and server may negotiate to choose
>>>>> proper compression protocol. This is why I prefer to perform
>>>>> negotiation between client and server to enable compression. Well,
>>>>> for negotiation you could put the name of the algorithm you want in
>>>>> the startup. It doesn't have to be a boolean for compression, and
>>>>> then you don't need an additional round-trip.
>>>> Sorry, I can only repeat arguments I already mentioned:
>>>> - in future it may be possible to specify compression algorithm
>>>> - even with boolean compression option server may have some reasons to
>>>> reject client's request to use compression
>>>>
>>>> Extra flexibility is always good thing if it doesn't cost too
>>>> much. And extra round of negotiation in case of enabling compression
>>>> seems to me not to be a high price for it.
>>> You already have this flexibility even without negotiation. I don't
>>> want you to lose your flexibility. Protocol looks like this:
>>>
>>> - Client sends connection option "compression" with list of
>>> algorithms it wants to use (comma-separated, or something).
>>>
>>> - First packet that the server can compress one of those algorithms
>>> (or none, if it doesn't want to turn on compression).
>>>
>>> No additional round-trips needed.
>> This is exactly how it works now... Client includes compression
>> option in connection string and server replies with special message
>> ('Z') if it accepts request to compress traffic between this client
>> and server.
> No, it's not. You don't need this message. If the server receives a
> compression request, it should just turn compression on (or not), and
> then have the client figure out whether it got compression back.
How it will managed to do it. It receives some reply and first of all it
should know whether it has to be decompressed or not.
> This
> is of course made harder by your refusal to use packet framing, but
> still shouldn't be particularly difficult.
But how?
>
> Thanks,
> --Robbie
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-22 15:59 ` Robbie Harwood <[email protected]>
2018-06-22 16:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-22 15:59 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 21.06.2018 20:14, Robbie Harwood wrote:
>> Konstantin Knizhnik <[email protected]> writes:
>>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>
>>>>>> Well, that's a design decision you've made. You could put
>>>>>> lengths on chunks that are sent out - then you'd know exactly how
>>>>>> much is needed. (For instance, 4 bytes of network-order length
>>>>>> followed by a complete payload.) Then you'd absolutely know
>>>>>> whether you have enough to decompress or not.
>>>>>
>>>>> Do you really suggest to send extra header for each chunk of data?
>>>>> Please notice that chunk can be as small as one message: dozen of
>>>>> bytes because libpq is used for client-server communication with
>>>>> request-reply pattern.
>>>>
>>>> I want you to think critically about your design. I *really* don't
>>>> want to design it for you - I have enough stuff to be doing. But
>>>> again, the design I gave you doesn't necessarily need that - you
>>>> just need to properly buffer incomplete data.
>>>
>>> Right now secure_read may return any number of available bytes. But
>>> in case of using streaming compression, it can happen that available
>>> number of bytes is not enough to perform decompression. This is why
>>> we may need to try to fetch additional portion of data. This is how
>>> zpq_stream is working now.
>>
>> No, you need to buffer and wait until you're called again. Which is
>> to say: decompress() shouldn't call secure_read(). secure_read()
>> should call decompress().
>
> I this case I will have to implement this code twice: both for backend
> and frontend, i.e. for secure_read/secure_write and
> pqsecure_read/pqsecure_write.
Likely, yes. You can see that this is how TLS does it (which you should
be using as a model, architecture-wise).
> Frankly speaking i was very upset by design of libpq communication
> layer in Postgres: there are two different implementations of almost
> the same stuff for cbackend and frontend.
Changing the codebases so that more could be shared is not necessarily a
bad idea; however, it is a separate change from compression.
>>> I do not understand how it is possible to implement in different way
>>> and what is wrong with current implementation.
>>
>> The most succinct thing I can say is: absolutely don't modify
>> pq_recvbuf(). I gave you pseudocode for how to do that. All of your
>> logic should be *below* the secure_read/secure_write functions.
>>
>> I cannot approve code that modifies pq_recvbuf() in the manner you
>> currently do.
>
> Well. I understand you arguments. But please also consider my
> argument above (about avoiding code duplication).
>
> In any case, secure_read function is called only from pq_recvbuf() as
> well as pqsecure_read is called only from pqReadData. So I do not
> think principle difference in handling compression in secure_read or
> pq_recvbuf functions and do not understand why it is "destroying the
> existing model".
>
> Frankly speaking, I will really like to destroy existed model, moving
> all system dependent stuff in Postgres to SAL and avoid this awful mix
> of code sharing and duplication between backend and frontend. But it
> is a another story and I do not want to discuss it here.
I understand you want to avoid code duplication. I will absolutely
agree that the current setup makes it difficult to share code between
postmaster and libpq clients. But the way I see it, you have two
choices:
1. Modify the code to make code sharing easier. Once this has been
done, *then* build a compression patch on top, with the nice new
architecture.
2. Leave the architecture as-is and add compression support.
(Optionally, you can make code sharing easier at a later point.)
Fundamentally, I think you value code sharing more than I do. So while
I might advocate for (2), you might personally prefer (1).
But right now you're not doing either of those.
> If we are speaking about the "right design", then neither your
> suggestion, neither my implementation are good and I do not see
> principle differences between them.
>
> The right approach is using "decorator pattern": this is how streams
> are designed in .Net/Java. You can construct pipe of "secure",
> "compressed" and whatever else streams.
I *strongly* disagree, but I don't think you're seriously suggesting
this.
>>>>>> My idea was the following: client want to use compression. But
>>>>>> server may reject this attempt (for any reasons: it doesn't support
>>>>>> it, has no proper compression library, do not want to spend CPU for
>>>>>> decompression,...) Right now compression algorithm is
>>>>>> hardcoded. But in future client and server may negotiate to choose
>>>>>> proper compression protocol. This is why I prefer to perform
>>>>>> negotiation between client and server to enable compression. Well,
>>>>>> for negotiation you could put the name of the algorithm you want in
>>>>>> the startup. It doesn't have to be a boolean for compression, and
>>>>>> then you don't need an additional round-trip.
>>>>> Sorry, I can only repeat arguments I already mentioned:
>>>>> - in future it may be possible to specify compression algorithm
>>>>> - even with boolean compression option server may have some reasons to
>>>>> reject client's request to use compression
>>>>>
>>>>> Extra flexibility is always good thing if it doesn't cost too
>>>>> much. And extra round of negotiation in case of enabling compression
>>>>> seems to me not to be a high price for it.
>>>> You already have this flexibility even without negotiation. I don't
>>>> want you to lose your flexibility. Protocol looks like this:
>>>>
>>>> - Client sends connection option "compression" with list of
>>>> algorithms it wants to use (comma-separated, or something).
>>>>
>>>> - First packet that the server can compress one of those algorithms
>>>> (or none, if it doesn't want to turn on compression).
>>>>
>>>> No additional round-trips needed.
>>> This is exactly how it works now... Client includes compression
>>> option in connection string and server replies with special message
>>> ('Z') if it accepts request to compress traffic between this client
>>> and server.
>>
>> No, it's not. You don't need this message. If the server receives a
>> compression request, it should just turn compression on (or not), and
>> then have the client figure out whether it got compression back.
>
> How it will managed to do it. It receives some reply and first of all
> it should know whether it has to be decompressed or not.
You can tell whether a message is compressed by looking at it. The way
the protocol works, every message has a type associated with it: a
single byte, like 'R', that says what kind of message it is.
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-22 15:59 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-22 16:32 ` Konstantin Knizhnik <[email protected]>
2018-06-22 17:56 ` Re: libpq compression Robbie Harwood <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-22 16:32 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 22.06.2018 18:59, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 21.06.2018 20:14, Robbie Harwood wrote:
>>> Konstantin Knizhnik <[email protected]> writes:
>>>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>>
>>>>>>> Well, that's a design decision you've made. You could put
>>>>>>> lengths on chunks that are sent out - then you'd know exactly how
>>>>>>> much is needed. (For instance, 4 bytes of network-order length
>>>>>>> followed by a complete payload.) Then you'd absolutely know
>>>>>>> whether you have enough to decompress or not.
>>>>>> Do you really suggest to send extra header for each chunk of data?
>>>>>> Please notice that chunk can be as small as one message: dozen of
>>>>>> bytes because libpq is used for client-server communication with
>>>>>> request-reply pattern.
>>>>> I want you to think critically about your design. I *really* don't
>>>>> want to design it for you - I have enough stuff to be doing. But
>>>>> again, the design I gave you doesn't necessarily need that - you
>>>>> just need to properly buffer incomplete data.
>>>> Right now secure_read may return any number of available bytes. But
>>>> in case of using streaming compression, it can happen that available
>>>> number of bytes is not enough to perform decompression. This is why
>>>> we may need to try to fetch additional portion of data. This is how
>>>> zpq_stream is working now.
>>> No, you need to buffer and wait until you're called again. Which is
>>> to say: decompress() shouldn't call secure_read(). secure_read()
>>> should call decompress().
>> I this case I will have to implement this code twice: both for backend
>> and frontend, i.e. for secure_read/secure_write and
>> pqsecure_read/pqsecure_write.
> Likely, yes. You can see that this is how TLS does it (which you should
> be using as a model, architecture-wise).
>
>> Frankly speaking i was very upset by design of libpq communication
>> layer in Postgres: there are two different implementations of almost
>> the same stuff for cbackend and frontend.
> Changing the codebases so that more could be shared is not necessarily a
> bad idea; however, it is a separate change from compression.
>
>>>> I do not understand how it is possible to implement in different way
>>>> and what is wrong with current implementation.
>>> The most succinct thing I can say is: absolutely don't modify
>>> pq_recvbuf(). I gave you pseudocode for how to do that. All of your
>>> logic should be *below* the secure_read/secure_write functions.
>>>
>>> I cannot approve code that modifies pq_recvbuf() in the manner you
>>> currently do.
>> Well. I understand you arguments. But please also consider my
>> argument above (about avoiding code duplication).
>>
>> In any case, secure_read function is called only from pq_recvbuf() as
>> well as pqsecure_read is called only from pqReadData. So I do not
>> think principle difference in handling compression in secure_read or
>> pq_recvbuf functions and do not understand why it is "destroying the
>> existing model".
>>
>> Frankly speaking, I will really like to destroy existed model, moving
>> all system dependent stuff in Postgres to SAL and avoid this awful mix
>> of code sharing and duplication between backend and frontend. But it
>> is a another story and I do not want to discuss it here.
> I understand you want to avoid code duplication. I will absolutely
> agree that the current setup makes it difficult to share code between
> postmaster and libpq clients. But the way I see it, you have two
> choices:
>
> 1. Modify the code to make code sharing easier. Once this has been
> done, *then* build a compression patch on top, with the nice new
> architecture.
>
> 2. Leave the architecture as-is and add compression support.
> (Optionally, you can make code sharing easier at a later point.)
>
> Fundamentally, I think you value code sharing more than I do. So while
> I might advocate for (2), you might personally prefer (1).
>
> But right now you're not doing either of those.
>
>> If we are speaking about the "right design", then neither your
>> suggestion, neither my implementation are good and I do not see
>> principle differences between them.
>>
>> The right approach is using "decorator pattern": this is how streams
>> are designed in .Net/Java. You can construct pipe of "secure",
>> "compressed" and whatever else streams.
> I *strongly* disagree, but I don't think you're seriously suggesting
> this.
>
>>>>>>> My idea was the following: client want to use compression. But
>>>>>>> server may reject this attempt (for any reasons: it doesn't support
>>>>>>> it, has no proper compression library, do not want to spend CPU for
>>>>>>> decompression,...) Right now compression algorithm is
>>>>>>> hardcoded. But in future client and server may negotiate to choose
>>>>>>> proper compression protocol. This is why I prefer to perform
>>>>>>> negotiation between client and server to enable compression. Well,
>>>>>>> for negotiation you could put the name of the algorithm you want in
>>>>>>> the startup. It doesn't have to be a boolean for compression, and
>>>>>>> then you don't need an additional round-trip.
>>>>>> Sorry, I can only repeat arguments I already mentioned:
>>>>>> - in future it may be possible to specify compression algorithm
>>>>>> - even with boolean compression option server may have some reasons to
>>>>>> reject client's request to use compression
>>>>>>
>>>>>> Extra flexibility is always good thing if it doesn't cost too
>>>>>> much. And extra round of negotiation in case of enabling compression
>>>>>> seems to me not to be a high price for it.
>>>>> You already have this flexibility even without negotiation. I don't
>>>>> want you to lose your flexibility. Protocol looks like this:
>>>>>
>>>>> - Client sends connection option "compression" with list of
>>>>> algorithms it wants to use (comma-separated, or something).
>>>>>
>>>>> - First packet that the server can compress one of those algorithms
>>>>> (or none, if it doesn't want to turn on compression).
>>>>>
>>>>> No additional round-trips needed.
>>>> This is exactly how it works now... Client includes compression
>>>> option in connection string and server replies with special message
>>>> ('Z') if it accepts request to compress traffic between this client
>>>> and server.
>>> No, it's not. You don't need this message. If the server receives a
>>> compression request, it should just turn compression on (or not), and
>>> then have the client figure out whether it got compression back.
>> How it will managed to do it. It receives some reply and first of all
>> it should know whether it has to be decompressed or not.
> You can tell whether a message is compressed by looking at it. The way
> the protocol works, every message has a type associated with it: a
> single byte, like 'R', that says what kind of message it is.
Compressed message can contain any sequence of bytes, including 'R':)
>
> Thanks,
> --Robbie
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-22 15:59 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 16:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-22 17:56 ` Robbie Harwood <[email protected]>
2018-06-23 11:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Robbie Harwood @ 2018-06-22 17:56 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Konstantin Knizhnik <[email protected]> writes:
> On 22.06.2018 18:59, Robbie Harwood wrote:
>> Konstantin Knizhnik <[email protected]> writes:
>>> On 21.06.2018 20:14, Robbie Harwood wrote:
>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>>>
>>>>>>>>> My idea was the following: client want to use compression. But
>>>>>>>>> server may reject this attempt (for any reasons: it doesn't
>>>>>>>>> support it, has no proper compression library, do not want to
>>>>>>>>> spend CPU for decompression,...) Right now compression
>>>>>>>>> algorithm is hardcoded. But in future client and server may
>>>>>>>>> negotiate to choose proper compression protocol. This is why
>>>>>>>>> I prefer to perform negotiation between client and server to
>>>>>>>>> enable compression.
>>>>>>>>
>>>>>>>> Well, for negotiation you could put the name of the algorithm
>>>>>>>> you want in the startup. It doesn't have to be a boolean for
>>>>>>>> compression, and then you don't need an additional round-trip.
>>>>>>>
>>>>>>> Sorry, I can only repeat arguments I already mentioned:
>>>>>>>
>>>>>>> - in future it may be possible to specify compression algorithm
>>>>>>>
>>>>>>> - even with boolean compression option server may have some
>>>>>>> reasons to reject client's request to use compression
>>>>>>>
>>>>>>> Extra flexibility is always good thing if it doesn't cost too
>>>>>>> much. And extra round of negotiation in case of enabling
>>>>>>> compression seems to me not to be a high price for it.
>>>>>>
>>>>>> You already have this flexibility even without negotiation. I
>>>>>> don't want you to lose your flexibility. Protocol looks like
>>>>>> this:
>>>>>>
>>>>>> - Client sends connection option "compression" with list of
>>>>>> algorithms it wants to use (comma-separated, or something).
>>>>>>
>>>>>> - First packet that the server can compress one of those algorithms
>>>>>> (or none, if it doesn't want to turn on compression).
>>>>>>
>>>>>> No additional round-trips needed.
>>>>>
>>>>> This is exactly how it works now... Client includes compression
>>>>> option in connection string and server replies with special
>>>>> message ('Z') if it accepts request to compress traffic between
>>>>> this client and server.
>>>>
>>>> No, it's not. You don't need this message. If the server receives
>>>> a compression request, it should just turn compression on (or not),
>>>> and then have the client figure out whether it got compression
>>>> back.
>>>
>>> How it will managed to do it. It receives some reply and first of
>>> all it should know whether it has to be decompressed or not.
>>
>> You can tell whether a message is compressed by looking at it. The
>> way the protocol works, every message has a type associated with it:
>> a single byte, like 'R', that says what kind of message it is.
>
> Compressed message can contain any sequence of bytes, including 'R':)
Then tag your messages with a type byte. Or do it the other way around
- look for the zstd framing within a message.
Please, try to work with me on this instead of fighting every design
change.
Thanks,
--Robbie
Attachments:
[application/pgp-signature] signature.asc (832B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-22 15:59 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-22 16:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-22 17:56 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-23 11:40 ` Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-23 11:40 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 22.06.2018 20:56, Robbie Harwood wrote:
> Konstantin Knizhnik <[email protected]> writes:
>
>> On 22.06.2018 18:59, Robbie Harwood wrote:
>>> Konstantin Knizhnik <[email protected]> writes:
>>>> On 21.06.2018 20:14, Robbie Harwood wrote:
>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>>>>>> Konstantin Knizhnik <[email protected]> writes:
>>>>>>>>>
>>>>>>>>>> My idea was the following: client want to use compression. But
>>>>>>>>>> server may reject this attempt (for any reasons: it doesn't
>>>>>>>>>> support it, has no proper compression library, do not want to
>>>>>>>>>> spend CPU for decompression,...) Right now compression
>>>>>>>>>> algorithm is hardcoded. But in future client and server may
>>>>>>>>>> negotiate to choose proper compression protocol. This is why
>>>>>>>>>> I prefer to perform negotiation between client and server to
>>>>>>>>>> enable compression.
>>>>>>>>> Well, for negotiation you could put the name of the algorithm
>>>>>>>>> you want in the startup. It doesn't have to be a boolean for
>>>>>>>>> compression, and then you don't need an additional round-trip.
>>>>>>>> Sorry, I can only repeat arguments I already mentioned:
>>>>>>>>
>>>>>>>> - in future it may be possible to specify compression algorithm
>>>>>>>>
>>>>>>>> - even with boolean compression option server may have some
>>>>>>>> reasons to reject client's request to use compression
>>>>>>>>
>>>>>>>> Extra flexibility is always good thing if it doesn't cost too
>>>>>>>> much. And extra round of negotiation in case of enabling
>>>>>>>> compression seems to me not to be a high price for it.
>>>>>>> You already have this flexibility even without negotiation. I
>>>>>>> don't want you to lose your flexibility. Protocol looks like
>>>>>>> this:
>>>>>>>
>>>>>>> - Client sends connection option "compression" with list of
>>>>>>> algorithms it wants to use (comma-separated, or something).
>>>>>>>
>>>>>>> - First packet that the server can compress one of those algorithms
>>>>>>> (or none, if it doesn't want to turn on compression).
>>>>>>>
>>>>>>> No additional round-trips needed.
>>>>>> This is exactly how it works now... Client includes compression
>>>>>> option in connection string and server replies with special
>>>>>> message ('Z') if it accepts request to compress traffic between
>>>>>> this client and server.
>>>>> No, it's not. You don't need this message. If the server receives
>>>>> a compression request, it should just turn compression on (or not),
>>>>> and then have the client figure out whether it got compression
>>>>> back.
>>>> How it will managed to do it. It receives some reply and first of
>>>> all it should know whether it has to be decompressed or not.
>>> You can tell whether a message is compressed by looking at it. The
>>> way the protocol works, every message has a type associated with it:
>>> a single byte, like 'R', that says what kind of message it is.
>> Compressed message can contain any sequence of bytes, including 'R':)
> Then tag your messages with a type byte. Or do it the other way around
> - look for the zstd framing within a message.
>
> Please, try to work with me on this instead of fighting every design
> change.
Sorry, I do not want fighting.
I am always vote for peace and constructive dialog.
But it is hard to me to understand and accept your arguments.
I do not understand why secure_read function is better place for
handling compression than pq_recvbuf.
And why it is destroying existed model.
I already mentioned my arguments:
1. I want to use the same code for frontend and backend.
2. I think that streaming compression can be used not only for libpq.
This is why I tried to make zpq_stream independent from
communication layer and pass here callbacks for sending/receiving data.
If pq_recvbuf is not right place for performing decommpression, I can
introduce some other function, like read_raw or something like that and
do decompression here. But I do not see much sense in it.
Concerning necessity of special message for acknowledging compression by
server: I also do not understand why you do not like idea to send some
message and what is wrong with it. What you are suggesting "then tag
your message" actually is the same as sending new message.
Because what is the difference between tag 'Z' and message with code 'Z'?
Sorry, but I do not understand problems you are going to solve and do
not see any arguments except "I can not accept it".
> Thanks,
> --Robbie
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-21 21:34 ` Nico Williams <[email protected]>
2018-06-22 07:18 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Nico Williams @ 2018-06-21 21:34 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote:
> On 20.06.2018 23:34, Robbie Harwood wrote:
> >Konstantin Knizhnik <[email protected]> writes:
> >Well, that's a design decision you've made. You could put lengths on
> >chunks that are sent out - then you'd know exactly how much is needed.
> >(For instance, 4 bytes of network-order length followed by a complete
> >payload.) Then you'd absolutely know whether you have enough to
> >decompress or not.
>
> Do you really suggest to send extra header for each chunk of data?
> Please notice that chunk can be as small as one message: dozen of bytes
> because libpq is used for client-server communication with request-reply
> pattern.
You must have lengths, yes, otherwise you're saying that the chosen
compression mechanism must itself provide framing.
I'm not that familiar with compression APIs and formats, but looking at
RFC1950 (zlib) for example I see no framing.
So I think you just have to have lengths.
Now, this being about compression, I understand that you might now want
to have 4-byte lengths, especially given that most messages will be
under 8KB. So use a varint encoding for the lengths.
Nico
--
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 21:34 ` Re: libpq compression Nico Williams <[email protected]>
@ 2018-06-22 07:18 ` Konstantin Knizhnik <[email protected]>
2018-06-22 16:05 ` Re: libpq compression Nico Williams <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-22 07:18 UTC (permalink / raw)
To: Nico Williams <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 22.06.2018 00:34, Nico Williams wrote:
> On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote:
>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>> Konstantin Knizhnik <[email protected]> writes:
>>> Well, that's a design decision you've made. You could put lengths on
>>> chunks that are sent out - then you'd know exactly how much is needed.
>>> (For instance, 4 bytes of network-order length followed by a complete
>>> payload.) Then you'd absolutely know whether you have enough to
>>> decompress or not.
>> Do you really suggest to send extra header for each chunk of data?
>> Please notice that chunk can be as small as one message: dozen of bytes
>> because libpq is used for client-server communication with request-reply
>> pattern.
> You must have lengths, yes, otherwise you're saying that the chosen
> compression mechanism must itself provide framing.
>
> I'm not that familiar with compression APIs and formats, but looking at
> RFC1950 (zlib) for example I see no framing.
>
> So I think you just have to have lengths.
>
> Now, this being about compression, I understand that you might now want
> to have 4-byte lengths, especially given that most messages will be
> under 8KB. So use a varint encoding for the lengths.
>
> Nico
No explicit framing and lengths are needed in case of using streaming
compression.
There can be certainly some kind of frames inside compression protocol
itself, but it is intrinsic of compression algorithm.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 21:34 ` Re: libpq compression Nico Williams <[email protected]>
2018-06-22 07:18 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-22 16:05 ` Nico Williams <[email protected]>
2018-06-22 16:35 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Nico Williams @ 2018-06-22 16:05 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote:
> On 22.06.2018 00:34, Nico Williams wrote:
> >So I think you just have to have lengths.
> >
> >Now, this being about compression, I understand that you might now want
> >to have 4-byte lengths, especially given that most messages will be
> >under 8KB. So use a varint encoding for the lengths.
>
> No explicit framing and lengths are needed in case of using streaming
> compression.
> There can be certainly some kind of frames inside compression protocol
> itself, but it is intrinsic of compression algorithm.
I don't think that's generally true. It may be true of the compression
algorithm you're working with. This is fine, of course, but plugging in
other compression algorithms will require the authors to add framing.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-21 21:34 ` Re: libpq compression Nico Williams <[email protected]>
2018-06-22 07:18 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-22 16:05 ` Re: libpq compression Nico Williams <[email protected]>
@ 2018-06-22 16:35 ` Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-22 16:35 UTC (permalink / raw)
To: Nico Williams <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 22.06.2018 19:05, Nico Williams wrote:
> On Fri, Jun 22, 2018 at 10:18:12AM +0300, Konstantin Knizhnik wrote:
>> On 22.06.2018 00:34, Nico Williams wrote:
>>> So I think you just have to have lengths.
>>>
>>> Now, this being about compression, I understand that you might now want
>>> to have 4-byte lengths, especially given that most messages will be
>>> under 8KB. So use a varint encoding for the lengths.
>> No explicit framing and lengths are needed in case of using streaming
>> compression.
>> There can be certainly some kind of frames inside compression protocol
>> itself, but it is intrinsic of compression algorithm.
> I don't think that's generally true. It may be true of the compression
> algorithm you're working with. This is fine, of course, but plugging in
> other compression algorithms will require the authors to add framing.
If compression algorithm supports streaming mode (and most of them
does), then you should not worry about frames.
And if compression algorithm doesn't support streaming mode, then it
should not be used for libpq traffic compression.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
@ 2018-06-25 09:32 ` Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Re: libpq compression Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-25 09:32 UTC (permalink / raw)
To: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 18.06.2018 23:34, Robbie Harwood wrote:
>
> ###
>
> Documentation! You're going to need it. There needs to be enough
> around for other people to implement the protocol (or if you prefer,
> enough for us to debug the protocol as it exists).
>
> In conjunction with that, please add information on how to set up
> compressed vs. uncompressed connections - similarly to how we've
> documentation on setting up TLS connection (though presumably compressed
> connection documentation will be shorter).
>
Document protocol changes needed for libpq compression.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] libpq-compression-7.patch (41.0K, ../../[email protected]/2-libpq-compression-7.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 498b8df..40c3187 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1115,6 +1115,17 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. If server is supporting compression, then all libpq messages send both from client to server and
+ visa versa will be compressed. Right now compression algorithm is hardcoded: is it is either zlib (default), either zstd (if Postgres was
+ configured with --with-zstd option). In both cases streaming mode is used.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index cfc805f..c01a51d 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ Is is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especialy useful for importing/exprorting data to/from database using COPY command
+ and for replication (oth physical and logical). Also compression can reduce server response time
+ in case of queries, requestion larger amount of data (for example returning JSON, BLOBs, text,...)
+ Right now compression algorithm is hardcoded: is it is either zlib (default), either zstd (if Postgres was
+ configured with --with-zstd option). In both cases streaming mode is used.
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -263,6 +272,18 @@
</varlistentry>
<varlistentry>
+ <term>CompressionOk</term>
+ <listitem>
+ <para>
+ Server acknowledge using compression for client-server communication protocol.
+ Compression can be requested by client by including "compression" option in connection string.
+ Right now compression algorithm is hardcoded, but in future client and server may negotiate to
+ choose proper compression algorithm.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term>AuthenticationOk</term>
<listitem>
<para>
@@ -3410,6 +3431,33 @@ AuthenticationSASLFinal (B)
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+CompressionOk (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data. After receiving this message bother server and client are switched to compression mode
+ and exchange compressed messages.
+</para>
+</listitem>
+
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
@@ -5826,6 +5874,19 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. Value can be
+ <literal>0</literal>, <literal>1</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>on</literal>, <literal>off.</literal>.
+ By default compression is disabled.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..c963657 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,31 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only conpression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression = 'z'; /* Request compression message */
+ int rc;
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, 1)) < 0 && errno == EINTR);
+ if (rc != 1)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +254,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +312,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +965,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +988,37 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ char const* msg = zpq_error(PqStream);
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1039,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1054,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1073,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1094,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1119,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1153,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1194,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1444,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1503,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f0c5149..efc4ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -188,6 +188,7 @@ int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
+bool libpq_compression; /* use libpq compression */
int main_pid; /* main process id used in log filename */
char *pghost = "";
@@ -590,6 +591,7 @@ usage(void)
" -h, --host=HOSTNAME database server host or socket directory\n"
" -p, --port=PORT database server port number\n"
" -U, --username=USERNAME connect as specified database user\n"
+ " -Z, --compression use libpq compression\n"
" -V, --version output version information, then exit\n"
" -?, --help show this help, then exit\n"
"\n"
@@ -1107,7 +1109,7 @@ doConnect(void)
*/
do
{
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
@@ -1124,8 +1126,10 @@ doConnect(void)
values[4] = dbName;
keywords[5] = "fallback_application_name";
values[5] = progname;
- keywords[6] = NULL;
- values[6] = NULL;
+ keywords[6] = "compression";
+ values[6] = libpq_compression ? "on" : "off";
+ keywords[7] = NULL;
+ values[7] = NULL;
new_pass = false;
@@ -4759,6 +4763,7 @@ main(int argc, char **argv)
{"builtin", required_argument, NULL, 'b'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
+ {"compression", no_argument, NULL, 'Z'},
{"debug", no_argument, NULL, 'd'},
{"define", required_argument, NULL, 'D'},
{"file", required_argument, NULL, 'f'},
@@ -4868,12 +4873,15 @@ main(int argc, char **argv)
exit(1);
}
- while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "iI:h:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:Z", long_options, &optindex)) != -1)
{
char *script;
switch (c)
{
+ case 'Z':
+ libpq_compression = true;
+ break;
case 'i':
is_init_mode = true;
break;
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index 702e742..ae7a14c 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -139,6 +139,7 @@ usage(unsigned short int pager)
fprintf(output, _(" -U, --username=USERNAME database user name (default: \"%s\")\n"), env);
fprintf(output, _(" -w, --no-password never prompt for password\n"));
fprintf(output, _(" -W, --password force password prompt (should happen automatically)\n"));
+ fprintf(output, _(" -Z, --compression compress traffic with server\n"));
fprintf(output, _("\nFor more information, type \"\\?\" (for internal commands) or \"\\help\" (for SQL\n"
"commands) from within psql, or consult the psql section in the PostgreSQL\n"
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
index be57574..9271716 100644
--- a/src/bin/psql/startup.c
+++ b/src/bin/psql/startup.c
@@ -75,6 +75,7 @@ struct adhoc_opts
bool no_psqlrc;
bool single_txn;
bool list_dbs;
+ bool compression;
SimpleActionList actions;
};
@@ -237,8 +238,10 @@ main(int argc, char *argv[])
values[5] = pset.progname;
keywords[6] = "client_encoding";
values[6] = (pset.notty || getenv("PGCLIENTENCODING")) ? NULL : "auto";
- keywords[7] = NULL;
- values[7] = NULL;
+ keywords[7] = "compression";
+ values[7] = options.compression ? "on" : "off";
+ keywords[8] = NULL;
+ values[8] = NULL;
new_pass = false;
pset.db = PQconnectdbParams(keywords, values, true);
@@ -436,6 +439,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
{"echo-all", no_argument, NULL, 'a'},
{"no-align", no_argument, NULL, 'A'},
{"command", required_argument, NULL, 'c'},
+ {"compression", no_argument, NULL, 'Z'},
{"dbname", required_argument, NULL, 'd'},
{"echo-queries", no_argument, NULL, 'e'},
{"echo-errors", no_argument, NULL, 'b'},
@@ -476,7 +480,7 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
memset(options, 0, sizeof *options);
- while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01",
+ while ((c = getopt_long(argc, argv, "aAbc:d:eEf:F:h:HlL:no:p:P:qR:sStT:U:v:VwWxXz?01Z",
long_options, &optindex)) != -1)
{
switch (c)
@@ -540,6 +544,9 @@ parse_psql_options(int argc, char *argv[], struct adhoc_opts *options)
case 'p':
options->port = pg_strdup(optarg);
break;
+ case 'Z':
+ options->compression = true;
+ break;
case 'P':
{
char *value;
diff --git a/src/bin/scripts/pg_isready.c b/src/bin/scripts/pg_isready.c
index f7ad7b4..5d035cd 100644
--- a/src/bin/scripts/pg_isready.c
+++ b/src/bin/scripts/pg_isready.c
@@ -34,7 +34,7 @@ main(int argc, char **argv)
const char *pghostaddr_str = NULL;
const char *pgport_str = NULL;
-#define PARAMS_ARRAY_SIZE 7
+#define PARAMS_ARRAY_SIZE 8
const char *keywords[PARAMS_ARRAY_SIZE];
const char *values[PARAMS_ARRAY_SIZE];
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..dbb05f2
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,368 @@
+#include "postgres_fe.h"
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx_error;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ int rc;
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ rc = deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ rc = inflateInit(&zs->rx);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx.msg;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+#else
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ return NULL;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size)
+{
+ return -1;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return NULL;
+}
+
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return 0;
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..dc765af
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,28 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..2dad4fb 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,16 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
@@ -49,7 +59,7 @@ endif
# src/backend/utils/mb
OBJS += encnames.o wchar.o
# src/common
-OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o
+OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS += fe-secure-openssl.o fe-secure-common.o sha2_openssl.o
@@ -106,7 +116,7 @@ backend_src = $(top_srcdir)/src/backend
chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c: % : $(top_srcdir)/src/port/%
rm -f $@ && $(LN_S) $< .
-ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c: % : $(top_srcdir)/src/common/%
+ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c: % : $(top_srcdir)/src/common/%
rm -f $@ && $(LN_S) $< .
encnames.c wchar.c: % : $(backend_src)/utils/mb/%
@@ -156,7 +166,7 @@ clean distclean: clean-lib
rm -f pg_config_paths.h
# Remove files we (may have) symlinked in from src/port and other places
rm -f chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c
- rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c
+ rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c
rm -f encnames.c wchar.c
maintainer-clean: distclean maintainer-clean-lib
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..f48ee0c 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -325,6 +326,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "COMPRESSION", "0", NULL,
+ "Libpq-compression", "Z", 1,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +435,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2657,23 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3483,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 593732f..ab2bad5 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -116,7 +116,7 @@ sub mkvcbuild
our @pgcommonallfiles = qw(
base64.c config_info.c controldata_utils.c exec.c file_perm.c ip.c
- keywords.c md5.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
+ keywords.c md5.c zpq_stream.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
saslprep.c scram-common.c string.c unicode_norm.c username.c
wait_error.c);
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-08-10 21:55 ` Andrew Dunstan <[email protected]>
2018-08-13 18:47 ` Re: libpq compression Robert Haas <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2018-08-10 21:55 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 06/25/2018 05:32 AM, Konstantin Knizhnik wrote:
>
>
> On 18.06.2018 23:34, Robbie Harwood wrote:
>>
>> ###
>>
>> Documentation! You're going to need it. There needs to be enough
>> around for other people to implement the protocol (or if you prefer,
>> enough for us to debug the protocol as it exists).
>>
>> In conjunction with that, please add information on how to set up
>> compressed vs. uncompressed connections - similarly to how we've
>> documentation on setting up TLS connection (though presumably compressed
>> connection documentation will be shorter).
>>
>
> Document protocol changes needed for libpq compression.
>
This thread appears to have gone quiet. What concerns me is that there
appears to be substantial disagreement between the author and the
reviewers. Since the last thing was this new patch it should really have
been put back into "needs review" (my fault to some extent - I missed
that). So rather than return the patch with feedfack I'm going to set it
to "needs review" and move it to the next CF. However, if we can't
arrive at a consensus about the direction during the next CF it should
be returned with feedback.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Re: libpq compression Andrew Dunstan <[email protected]>
@ 2018-08-13 18:47 ` Robert Haas <[email protected]>
2018-08-13 20:06 ` Re: libpq compression Andrew Dunstan <[email protected]>
2018-08-14 14:50 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Robert Haas @ 2018-08-13 18:47 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan
<[email protected]> wrote:
> This thread appears to have gone quiet. What concerns me is that there
> appears to be substantial disagreement between the author and the reviewers.
> Since the last thing was this new patch it should really have been put back
> into "needs review" (my fault to some extent - I missed that). So rather
> than return the patch with feedfack I'm going to set it to "needs review"
> and move it to the next CF. However, if we can't arrive at a consensus about
> the direction during the next CF it should be returned with feedback.
I agree with the critiques from Robbie Harwood and Michael Paquier
that the way in that compression is being hooked into the existing
architecture looks like a kludge. I'm not sure I know exactly how it
should be done, but the current approach doesn't look natural; it
looks like it was bolted on. I agree with the critique from Peter
Eisentraut and others that we should not go around and add -Z as a
command-line option to all of our tools; this feature hasn't yet
proved itself to be useful enough to justify that. Better to let
people specify it via a connection string option if they want it. I
think Thomas Munro was right to ask about what will happen when
different compression libraries are in use, and I think failing
uncleanly is quite unacceptable. I think there needs to be some
method for negotiating the compression type; the client can, for
example, provide a comma-separated list of methods it supports in
preference order, and the server can pick the first one it likes. In
short, I think that a number of people have provided really good
feedback on this patch, and I suggest to Konstantin that he should
consider accepting all of those suggestions.
Commit ae65f6066dc3d19a55f4fdcd3b30003c5ad8dbed tried to introduce
some facilities that can be used for protocol version negotiation as
new features are added, but this patch doesn't use them. It looks to
me like it instead just breaks backward compatibility. The new
'compression' option won't be recognized by older servers. If it were
spelled '_pq_.compression' then the facilities in that commit would
cause a NegotiateProtocolVersion message to be sent by servers which
have that commit but don't support the compression option. I'm not
exactly sure what will happen on even-older servers that don't have
that commit either, but I hope they'll treat it as a GUC name; note
that setting an unknown GUC name with a namespace prefix is not an
error, but setting one without such a prefix IS an ERROR. Servers
which do support compression would respond with a message indicating
that compression had been enabled or, maybe, just start sending back
compressed-packet messages, if we go with including some framing in
the libpq protocol.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Re: libpq compression Andrew Dunstan <[email protected]>
2018-08-13 18:47 ` Re: libpq compression Robert Haas <[email protected]>
@ 2018-08-13 20:06 ` Andrew Dunstan <[email protected]>
2018-08-20 15:00 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2018-08-13 20:06 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 08/13/2018 02:47 PM, Robert Haas wrote:
> On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan
> <[email protected]> wrote:
>> This thread appears to have gone quiet. What concerns me is that there
>> appears to be substantial disagreement between the author and the reviewers.
>> Since the last thing was this new patch it should really have been put back
>> into "needs review" (my fault to some extent - I missed that). So rather
>> than return the patch with feedfack I'm going to set it to "needs review"
>> and move it to the next CF. However, if we can't arrive at a consensus about
>> the direction during the next CF it should be returned with feedback.
> I agree with the critiques from Robbie Harwood and Michael Paquier
> that the way in that compression is being hooked into the existing
> architecture looks like a kludge. I'm not sure I know exactly how it
> should be done, but the current approach doesn't look natural; it
> looks like it was bolted on. I agree with the critique from Peter
> Eisentraut and others that we should not go around and add -Z as a
> command-line option to all of our tools; this feature hasn't yet
> proved itself to be useful enough to justify that. Better to let
> people specify it via a connection string option if they want it. I
> think Thomas Munro was right to ask about what will happen when
> different compression libraries are in use, and I think failing
> uncleanly is quite unacceptable. I think there needs to be some
> method for negotiating the compression type; the client can, for
> example, provide a comma-separated list of methods it supports in
> preference order, and the server can pick the first one it likes. In
> short, I think that a number of people have provided really good
> feedback on this patch, and I suggest to Konstantin that he should
> consider accepting all of those suggestions.
>
> Commit ae65f6066dc3d19a55f4fdcd3b30003c5ad8dbed tried to introduce
> some facilities that can be used for protocol version negotiation as
> new features are added, but this patch doesn't use them. It looks to
> me like it instead just breaks backward compatibility. The new
> 'compression' option won't be recognized by older servers. If it were
> spelled '_pq_.compression' then the facilities in that commit would
> cause a NegotiateProtocolVersion message to be sent by servers which
> have that commit but don't support the compression option. I'm not
> exactly sure what will happen on even-older servers that don't have
> that commit either, but I hope they'll treat it as a GUC name; note
> that setting an unknown GUC name with a namespace prefix is not an
> error, but setting one without such a prefix IS an ERROR. Servers
> which do support compression would respond with a message indicating
> that compression had been enabled or, maybe, just start sending back
> compressed-packet messages, if we go with including some framing in
> the libpq protocol.
>
Excellent summary, and well argued recommendations, thanks. I've changed
the status to waiting on author.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Re: libpq compression Andrew Dunstan <[email protected]>
2018-08-13 18:47 ` Re: libpq compression Robert Haas <[email protected]>
2018-08-13 20:06 ` Re: libpq compression Andrew Dunstan <[email protected]>
@ 2018-08-20 15:00 ` Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-08-20 15:00 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; Robert Haas <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 13.08.2018 23:06, Andrew Dunstan wrote:
>
>
> On 08/13/2018 02:47 PM, Robert Haas wrote:
>> On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan
>> <[email protected]> wrote:
>>> This thread appears to have gone quiet. What concerns me is that there
>>> appears to be substantial disagreement between the author and the
>>> reviewers.
>>> Since the last thing was this new patch it should really have been
>>> put back
>>> into "needs review" (my fault to some extent - I missed that). So
>>> rather
>>> than return the patch with feedfack I'm going to set it to "needs
>>> review"
>>> and move it to the next CF. However, if we can't arrive at a
>>> consensus about
>>> the direction during the next CF it should be returned with feedback.
>> I agree with the critiques from Robbie Harwood and Michael Paquier
>> that the way in that compression is being hooked into the existing
>> architecture looks like a kludge. I'm not sure I know exactly how it
>> should be done, but the current approach doesn't look natural; it
>> looks like it was bolted on. I agree with the critique from Peter
>> Eisentraut and others that we should not go around and add -Z as a
>> command-line option to all of our tools; this feature hasn't yet
>> proved itself to be useful enough to justify that. Better to let
>> people specify it via a connection string option if they want it. I
>> think Thomas Munro was right to ask about what will happen when
>> different compression libraries are in use, and I think failing
>> uncleanly is quite unacceptable. I think there needs to be some
>> method for negotiating the compression type; the client can, for
>> example, provide a comma-separated list of methods it supports in
>> preference order, and the server can pick the first one it likes. In
>> short, I think that a number of people have provided really good
>> feedback on this patch, and I suggest to Konstantin that he should
>> consider accepting all of those suggestions.
>>
>> Commit ae65f6066dc3d19a55f4fdcd3b30003c5ad8dbed tried to introduce
>> some facilities that can be used for protocol version negotiation as
>> new features are added, but this patch doesn't use them. It looks to
>> me like it instead just breaks backward compatibility. The new
>> 'compression' option won't be recognized by older servers. If it were
>> spelled '_pq_.compression' then the facilities in that commit would
>> cause a NegotiateProtocolVersion message to be sent by servers which
>> have that commit but don't support the compression option. I'm not
>> exactly sure what will happen on even-older servers that don't have
>> that commit either, but I hope they'll treat it as a GUC name; note
>> that setting an unknown GUC name with a namespace prefix is not an
>> error, but setting one without such a prefix IS an ERROR. Servers
>> which do support compression would respond with a message indicating
>> that compression had been enabled or, maybe, just start sending back
>> compressed-packet messages, if we go with including some framing in
>> the libpq protocol.
>>
>
>
> Excellent summary, and well argued recommendations, thanks. I've
> changed the status to waiting on author.
>
New version of the patch is attached: I removed -Z options form pgbench
and psql and add checking that server and client are implementing the
same compression algorithm.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] libpq-compression-8.patch (36.7K, ../../[email protected]/2-libpq-compression-8.patch)
download | inline diff:
diff --git a/configure b/configure
index 0aafd9c..fc5685c 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,7 @@ ELF_SYS
EGREP
GREP
with_zlib
+with_zstd
with_system_tzdata
with_libxslt
with_libxml
@@ -863,6 +864,7 @@ with_libxml
with_libxslt
with_system_tzdata
with_zlib
+with_zstd
with_gnu_ld
enable_largefile
enable_float4_byval
@@ -8017,6 +8019,86 @@ fi
#
+# ZStd
+#
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+
+
+if test "$with_zstd" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compress in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compress in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compress+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compress ();
+int
+main ()
+{
+return ZSTD_compress ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_compress=yes
+else
+ ac_cv_lib_zstd_ZSTD_compress=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compress" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compress" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compress" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "library 'zstd' is required for ZSTD support" "$LINENO" 5
+fi
+
+fi
+
+
+
+#
# Elf
#
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 498b8df..40c3187 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1115,6 +1115,17 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. If server is supporting compression, then all libpq messages send both from client to server and
+ visa versa will be compressed. Right now compression algorithm is hardcoded: is it is either zlib (default), either zstd (if Postgres was
+ configured with --with-zstd option). In both cases streaming mode is used.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index cfc805f..c328dd7 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ Is is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especialy useful for importing/exprorting data to/from database using COPY command
+ and for replication (oth physical and logical). Also compression can reduce server response time
+ in case of queries, requestion larger amount of data (for example returning JSON, BLOBs, text,...)
+ Right now compression algorithm is hardcoded: is it is either zlib (default), either zstd (if Postgres was
+ configured with --with-zstd option). In both cases streaming mode is used.
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -263,6 +272,18 @@
</varlistentry>
<varlistentry>
+ <term>CompressionOk</term>
+ <listitem>
+ <para>
+ Server acknowledge using compression for client-server communication protocol.
+ Compression can be requested by client by including "compression" option in connection string.
+ Right now compression algorithm is hardcoded, but in future client and server may negotiate to
+ choose proper compression algorithm.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term>AuthenticationOk</term>
<listitem>
<para>
@@ -3410,6 +3431,43 @@ AuthenticationSASLFinal (B)
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+CompressionOk (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data. After receiving this message bother server and client are switched to compression mode
+ and exchange compressed messages.
+</para>
+</listitem>
+
+</varlistentry>
+<varlistentry>
+<term>
+ Byte1
+</term>
+<listitem>
+<para>
+ Used compression algorithm. Right now the following streaming compression algorithms are supported: 'f' - Facebook zstd, 'z' - zlib.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
@@ -5826,6 +5884,19 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. Value can be
+ <literal>0</literal>, <literal>1</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>on</literal>, <literal>off.</literal>.
+ By default compression is disabled.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 95d090e..b59629c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 25af514..a8e461a 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -51,6 +51,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index a4f6d4d..d48fdd0 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -95,6 +95,7 @@
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -143,6 +144,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream* PqStream;
+
+
/*
* Message status
*/
@@ -185,6 +189,34 @@ PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port* port)
+{
+ if (port->use_compression)
+ {
+ char compression[2];
+ int rc;
+ compression[0] = 'z'; /* Request compression message */
+ compression[1] = zpq_algorithm();
+ /* Switch on compression at client side */
+ socket_set_nonblocking(false);
+ while ((rc = secure_write(MyProcPort, &compression, sizeof compression)) < 0
+ && errno == EINTR);
+ if (rc != 2)
+ return -1;
+
+ /* initialize compression */
+ PqStream = zpq_create((zpq_tx_func)secure_write, (zpq_rx_func)secure_read, MyProcPort);
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -225,6 +257,7 @@ pq_init(void)
NULL, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_LATCH_SET, -1, MyLatch, NULL);
AddWaitEventToSet(FeBeWaitSet, WL_POSTMASTER_DEATH, -1, NULL, NULL);
+
}
/* --------------------------------
@@ -282,6 +315,9 @@ socket_close(int code, Datum arg)
free(MyProcPort->gss);
#endif /* ENABLE_GSS || ENABLE_SSPI */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -932,12 +968,14 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -953,21 +991,37 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ size_t processed = 0;
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, &processed)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ PqRecvLength += processed;
if (r < 0)
{
+ if (r == ZPQ_DECOMPRESS_ERROR)
+ {
+ char const* msg = zpq_error(PqStream);
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -988,7 +1042,7 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ return r;
}
}
@@ -1003,7 +1057,7 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1022,7 +1076,7 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1043,44 +1097,11 @@ pq_getbyte_if_available(unsigned char *c)
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1101,7 +1122,7 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1135,7 +1156,7 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1176,7 +1197,7 @@ pq_getstring(StringInfo s)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv some */
return EOF; /* Failed to recv data */
}
@@ -1426,13 +1447,18 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered(PqStream) != 0) /* has more data to flush or unsent data in internal compression buffer */
{
- int r;
-
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
-
- if (r <= 0)
+ int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
+
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1480,7 +1506,6 @@ internal_flush(void)
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index a4b53b3..3928e89 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2053,6 +2053,16 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "compression") == 0)
+ {
+ if (!parse_bool(valptr, &port->use_compression))
+ ereport(FATAL,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid boolean value for parameter \"%s\": \"%s\"",
+ "compression",
+ valptr),
+ errhint("Valid values are: \"false\", \"off\", 0, \"true\", \"on\", 1.")));
+ }
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4257,6 +4267,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/common/Makefile b/src/common/Makefile
index 1fc2c66..f804fe1 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -43,7 +43,7 @@ LIBS += $(PTHREAD_LIBS)
OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
ip.o keywords.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o \
rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
- username.o wait_error.o
+ username.o wait_error.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS_COMMON += sha2_openssl.o
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000..afd42e9
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,386 @@
+#include "postgres_fe.h"
+#include "common/zpq_stream.h"
+#include "c.h"
+#include "pg_config.h"
+
+#if HAVE_LIBZSTD
+
+#include <malloc.h>
+#include <zstd.h>
+
+#define ZPQ_BUFFER_SIZE (8*1024)
+#define ZSTD_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ ZSTD_CStream* tx_stream;
+ ZSTD_DStream* rx_stream;
+ ZSTD_outBuffer tx;
+ ZSTD_inBuffer rx;
+ size_t tx_not_flushed; /* Amount of datas in internal zstd buffer */
+ size_t tx_buffered; /* Data which is consumed by zpq_read but not yet sent */
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+ char const* rx_error; /* Decompress error message */
+ size_t tx_total;
+ size_t tx_total_raw;
+ size_t rx_total;
+ size_t rx_total_raw;
+ char tx_buf[ZPQ_BUFFER_SIZE];
+ char rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ zs->tx_stream = ZSTD_createCStream();
+ ZSTD_initCStream(zs->tx_stream, ZSTD_COMPRESSION_LEVEL);
+ zs->rx_stream = ZSTD_createDStream();
+ ZSTD_initDStream(zs->rx_stream);
+ zs->tx.dst = zs->tx_buf;
+ zs->tx.pos = 0;
+ zs->tx.size = ZPQ_BUFFER_SIZE;
+ zs->rx.src = zs->rx_buf;
+ zs->rx.pos = 0;
+ zs->rx.size = 0;
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->tx_buffered = 0;
+ zs->tx_not_flushed = 0;
+ zs->rx_error = NULL;
+ zs->arg = arg;
+ zs->tx_total = zs->tx_total_raw = 0;
+ zs->rx_total = zs->rx_total_raw = 0;
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_outBuffer out;
+ out.dst = buf;
+ out.pos = 0;
+ out.size = size;
+
+ while (1)
+ {
+ rc = ZSTD_decompressStream(zs->rx_stream, &out, &zs->rx);
+ if (ZSTD_isError(rc))
+ {
+ zs->rx_error = ZSTD_getErrorName(rc);
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ /* Return result if we fill requested amount of bytes or read operation was performed */
+ if (out.pos != 0)
+ {
+ zs->rx_total_raw += out.pos;
+ return out.pos;
+ }
+ if (zs->rx.pos == zs->rx.size)
+ {
+ zs->rx.pos = zs->rx.size = 0; /* Reset rx buffer */
+ }
+ rc = zs->rx_func(zs->arg, (char*)zs->rx.src + zs->rx.size, ZPQ_BUFFER_SIZE - zs->rx.size);
+ if (rc > 0) /* read fetches some data */
+ {
+ zs->rx.size += rc;
+ zs->rx_total += rc;
+ }
+ else /* read failed */
+ {
+ *processed = out.pos;
+ zs->rx_total_raw += out.pos;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ ssize_t rc;
+ ZSTD_inBuffer in_buf;
+ in_buf.src = buf;
+ in_buf.pos = 0;
+ in_buf.size = size;
+
+ do
+ {
+ if (zs->tx.pos == 0) /* Compress buffer is empty */
+ {
+ zs->tx.dst = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (in_buf.pos < size) /* Has something to compress in input buffer */
+ ZSTD_compressStream(zs->tx_stream, &zs->tx, &in_buf);
+
+ if (in_buf.pos == size) /* All data is compressed: flushed internal zstd buffer */
+ {
+ zs->tx_not_flushed = ZSTD_flushStream(zs->tx_stream, &zs->tx);
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.dst, zs->tx.pos);
+ if (rc > 0)
+ {
+ zs->tx.pos -= rc;
+ zs->tx.dst = (char*)zs->tx.dst + rc;
+ zs->tx_total += rc;
+ }
+ else
+ {
+ *processed = in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ zs->tx_total_raw += in_buf.pos;
+ return rc;
+ }
+ } while (zs->tx.pos == 0 && (in_buf.pos < size || zs->tx_not_flushed)); /* repeat sending data until first partial write */
+
+ zs->tx_total_raw += in_buf.pos;
+ zs->tx_buffered = zs->tx.pos;
+ return in_buf.pos;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ ZSTD_freeCStream(zs->tx_stream);
+ ZSTD_freeDStream(zs->rx_stream);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx_error;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered + zs->tx_not_flushed : 0;
+}
+
+char
+zpq_algorithm(void)
+{
+ return 'f';
+}
+
+#elif HAVE_LIBZ
+
+#include <malloc.h>
+#include <zlib.h>
+
+#define ZPQ_BUFFER_SIZE 8192
+#define ZLIB_COMPRESSION_LEVEL 1
+
+struct ZpqStream
+{
+ z_stream tx;
+ z_stream rx;
+
+ zpq_tx_func tx_func;
+ zpq_rx_func rx_func;
+ void* arg;
+
+ size_t tx_buffered;
+
+ Bytef tx_buf[ZPQ_BUFFER_SIZE];
+ Bytef rx_buf[ZPQ_BUFFER_SIZE];
+};
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ int rc;
+ ZpqStream* zs = (ZpqStream*)malloc(sizeof(ZpqStream));
+ memset(&zs->tx, 0, sizeof(zs->tx));
+ zs->tx.next_out = zs->tx_buf;
+ zs->tx.avail_out = ZPQ_BUFFER_SIZE;
+ zs->tx_buffered = 0;
+ rc = deflateInit(&zs->tx, ZLIB_COMPRESSION_LEVEL);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->tx.next_out == zs->tx_buf && zs->tx.avail_out == ZPQ_BUFFER_SIZE);
+
+ memset(&zs->rx, 0, sizeof(zs->tx));
+ zs->rx.next_in = zs->rx_buf;
+ zs->rx.avail_in = ZPQ_BUFFER_SIZE;
+ rc = inflateInit(&zs->rx);
+ if (rc != Z_OK)
+ {
+ free(zs);
+ return NULL;
+ }
+ Assert(zs->rx.next_in == zs->rx_buf && zs->rx.avail_in == ZPQ_BUFFER_SIZE);
+ zs->rx.avail_in = 0;
+
+ zs->rx_func = rx_func;
+ zs->tx_func = tx_func;
+ zs->arg = arg;
+
+ return zs;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->rx.next_out = (Bytef *)buf;
+ zs->rx.avail_out = size;
+
+ while (1)
+ {
+ if (zs->rx.avail_in != 0) /* If there is some data in receiver buffer, then decompress it */
+ {
+ rc = inflate(&zs->rx, Z_SYNC_FLUSH);
+ if (rc != Z_OK)
+ {
+ return ZPQ_DECOMPRESS_ERROR;
+ }
+ if (zs->rx.avail_out != size)
+ {
+ return size - zs->rx.avail_out;
+ }
+ if (zs->rx.avail_in == 0)
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ }
+ else
+ {
+ zs->rx.next_in = zs->rx_buf;
+ }
+ rc = zs->rx_func(zs->arg, zs->rx.next_in + zs->rx.avail_in, zs->rx_buf + ZPQ_BUFFER_SIZE - zs->rx.next_in - zs->rx.avail_in);
+ if (rc > 0)
+ {
+ zs->rx.avail_in += rc;
+ }
+ else
+ {
+ *processed = size - zs->rx.avail_out;
+ return rc;
+ }
+ }
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size, size_t *processed)
+{
+ int rc;
+ zs->tx.next_in = (Bytef *)buf;
+ zs->tx.avail_in = size;
+ do
+ {
+ if (zs->tx.avail_out == ZPQ_BUFFER_SIZE) /* Compress buffer is empty */
+ {
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+
+ if (zs->tx.avail_in != 0) /* Has something in input buffer */
+ {
+ rc = deflate(&zs->tx, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ zs->tx.next_out = zs->tx_buf; /* Reset pointer to the beginning of buffer */
+ }
+ }
+ rc = zs->tx_func(zs->arg, zs->tx.next_out, ZPQ_BUFFER_SIZE - zs->tx.avail_out);
+ if (rc > 0)
+ {
+ zs->tx.next_out += rc;
+ zs->tx.avail_out += rc;
+ }
+ else
+ {
+ *processed = size - zs->tx.avail_in;
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+ return rc;
+ }
+ } while (zs->tx.avail_out == ZPQ_BUFFER_SIZE && zs->tx.avail_in != 0); /* repeat sending data until first partial write */
+
+ zs->tx_buffered = ZPQ_BUFFER_SIZE - zs->tx.avail_out;
+
+ return size - zs->tx.avail_in;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+ if (zs != NULL)
+ {
+ inflateEnd(&zs->rx);
+ deflateEnd(&zs->tx);
+ free(zs);
+ }
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return zs->rx.msg;
+}
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return zs != NULL ? zs->tx_buffered : 0;
+}
+
+char
+zpq_algorithm(void)
+{
+ return 'z';
+}
+
+#else
+
+ZpqStream*
+zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg)
+{
+ return NULL;
+}
+
+ssize_t
+zpq_read(ZpqStream *zs, void *buf, size_t size)
+{
+ return -1;
+}
+
+ssize_t
+zpq_write(ZpqStream *zs, void const *buf, size_t size)
+{
+ return -1;
+}
+
+void
+zpq_free(ZpqStream *zs)
+{
+}
+
+char const*
+zpq_error(ZpqStream *zs)
+{
+ return NULL;
+}
+
+
+size_t
+zpq_buffered(ZpqStream *zs)
+{
+ return 0;
+}
+
+char
+zpq_algorithm(void)
+{
+ return '0';
+}
+
+#endif
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000..30dc98d
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,29 @@
+/*
+ * zpq_stream.h
+ * Streaiming compression for libpq
+ */
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#include <stdlib.h>
+
+#define ZPQ_IO_ERROR (-1)
+#define ZPQ_DECOMPRESS_ERROR (-2)
+
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t(*zpq_tx_func)(void* arg, void const* data, size_t size);
+typedef ssize_t(*zpq_rx_func)(void* arg, void* data, size_t size);
+
+
+ZpqStream* zpq_create(zpq_tx_func tx_func, zpq_rx_func rx_func, void* arg);
+ssize_t zpq_read(ZpqStream* zs, void* buf, size_t size, size_t* processed);
+ssize_t zpq_write(ZpqStream* zs, void const* buf, size_t size, size_t* processed);
+char const* zpq_error(ZpqStream* zs);
+size_t zpq_buffered(ZpqStream* zs);
+void zpq_free(ZpqStream* zs);
+char zpq_algorithm(void);
+
+#endif
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7698cd1..5203f2d 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -182,6 +182,8 @@ typedef struct Port
char *peer_cn;
bool peer_cert_valid;
+ bool use_compression;
+
/*
* OpenSSL structures. (Keep these last so that the locations of other
* fields are the same whether or not you build with OpenSSL.)
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 7bf06c6..cb0b69e 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -61,6 +61,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port* port);
extern int pq_getbytes(char *s, size_t len);
extern int pq_getstring(StringInfo s);
extern void pq_startmsgread(void);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9411f48..c0ea383 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -365,6 +365,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if the system has the type `locale_t'. */
#undef HAVE_LOCALE_T
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index abe0a50..2dad4fb 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,16 @@ endif
# platforms require special flags.
LIBS := $(LIBS:-lpgport=)
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
# We can't use Makefile variables here because the MSVC build system scrapes
# OBJS from this file.
OBJS= fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \
@@ -49,7 +59,7 @@ endif
# src/backend/utils/mb
OBJS += encnames.o wchar.o
# src/common
-OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o
+OBJS += base64.o ip.o md5.o scram-common.o saslprep.o unicode_norm.o zpq_stream.o
ifeq ($(with_openssl),yes)
OBJS += fe-secure-openssl.o fe-secure-common.o sha2_openssl.o
@@ -106,7 +116,7 @@ backend_src = $(top_srcdir)/src/backend
chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c: % : $(top_srcdir)/src/port/%
rm -f $@ && $(LN_S) $< .
-ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c: % : $(top_srcdir)/src/common/%
+ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c: % : $(top_srcdir)/src/common/%
rm -f $@ && $(LN_S) $< .
encnames.c wchar.c: % : $(backend_src)/utils/mb/%
@@ -156,7 +166,7 @@ clean distclean: clean-lib
rm -f pg_config_paths.h
# Remove files we (may have) symlinked in from src/port and other places
rm -f chklocale.c crypt.c erand48.c getaddrinfo.c getpeereid.c inet_aton.c inet_net_ntop.c noblock.c open.c system.c pgsleep.c pg_strong_random.c pgstrcasecmp.c pqsignal.c snprintf.c strerror.c strlcpy.c strnlen.c thread.c win32error.c win32setlocale.c
- rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c
+ rm -f ip.c md5.c base64.c scram-common.c sha2.c sha2_openssl.c saslprep.c unicode_norm.c zpq_stream.c
rm -f encnames.c wchar.c
maintainer-clean: distclean maintainer-clean-lib
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a7e969d..029a982 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -72,6 +72,7 @@ static int ldapServiceLookup(const char *purl, PQconninfoOption *options,
#include "common/ip.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "mb/pg_wchar.h"
#include "port/pg_bswap.h"
@@ -325,6 +326,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "COMPRESSION", NULL, NULL,
+ "Libpq-compression", "Z", 1,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 11, /* sizeof("read-write") = 11 */
@@ -430,6 +435,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zstream);
+ conn->zstream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -2648,11 +2657,33 @@ keep_going: /* We will come back to here until there is
*/
conn->inCursor = conn->inStart;
- /* Read type byte */
- if (pqGetc(&beresp, conn))
+ while (1)
{
- /* We'll come back when there is more data */
- return PGRES_POLLING_READING;
+ /* Read type byte */
+ if (pqGetc(&beresp, conn))
+ {
+ /* We'll come back when there is more data */
+ return PGRES_POLLING_READING;
+ }
+
+ if (beresp == 'z') /* Switch on compression */
+ {
+ char algorithm;
+ pqGetc(&algorithm, conn);
+ if (zpq_algorithm() != algorithm)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext(
+ "server and client were configured with different libpq compression algorithms: %c vs. %c\n"),
+ algorithm, zpq_algorithm());
+ goto error_return;
+ }
+ /* mark byte consumed */
+ conn->inStart = conn->inCursor;
+ Assert(!conn->zstream);
+ conn->zstream = zpq_create((zpq_tx_func)pqsecure_write, (zpq_rx_func)pqsecure_read, conn);
+ } else
+ break;
}
/*
@@ -3462,6 +3493,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 2a6637f..488f8d2 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,11 +53,12 @@
#include "port/pg_bswap.h"
#include "pg_config_paths.h"
+#include <common/zpq_stream.h>
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
-static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
- time_t end_time);
+static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
+ time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
/*
@@ -630,6 +631,7 @@ pqReadData(PGconn *conn)
{
int someread = 0;
int nread;
+ size_t processed;
if (conn->sock == PGINVALID_SOCKET)
{
@@ -678,10 +680,23 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry3;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -768,10 +783,24 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ processed = 0;
+ nread = conn->zstream
+ ? zpq_read(conn->zstream, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd, &processed)
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd,
+ conn->inBufSize - conn->inEnd);
+ conn->inEnd += processed;
+
if (nread < 0)
{
+ if (nread == ZPQ_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_error(conn->zstream));
+ return -1;
+ }
+
if (SOCK_ERRNO == EINTR)
goto retry4;
/* Some systems return EAGAIN/EWOULDBLOCK for no data */
@@ -842,12 +871,14 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered(conn->zstream))
{
int sent;
-
+ size_t processed = 0;
+ sent = conn->zstream
+ ? zpq_write(conn->zstream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -855,8 +886,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+ : pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +930,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered(conn->zstream))
{
/*
* We didn't send it all, wait till we can send more.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 8345faf..3942be1 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2179,6 +2179,8 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("database", conn->dbName);
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
if (conn->send_appname)
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 9a586ff..6344eb6 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "pqexpbuffer.h"
@@ -357,6 +358,7 @@ struct pg_conn
char *sslrootcert; /* root certificate filename */
char *sslcrl; /* certificate revocation list filename */
char *requirepeer; /* required peer credentials for local sockets */
+ char *compression; /* stream compression (0 or 1) */
#if defined(ENABLE_GSS) || defined(ENABLE_SSPI)
char *krbsrvname; /* Kerberos service name */
@@ -495,6 +497,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream* zstream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 593732f..ab2bad5 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -116,7 +116,7 @@ sub mkvcbuild
our @pgcommonallfiles = qw(
base64.c config_info.c controldata_utils.c exec.c file_perm.c ip.c
- keywords.c md5.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
+ keywords.c md5.c zpq_stream.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
saslprep.c scram-common.c string.c unicode_norm.c username.c
wait_error.c);
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 14:06 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-06 09:37 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Re: libpq compression Robbie Harwood <[email protected]>
2018-06-25 09:32 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Re: libpq compression Andrew Dunstan <[email protected]>
2018-08-13 18:47 ` Re: libpq compression Robert Haas <[email protected]>
@ 2018-08-14 14:50 ` Konstantin Knizhnik <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-08-14 14:50 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: Robbie Harwood <[email protected]>; Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Robert,
First of all thank you for review and your time spent on analyzing this
patch.
My comments are inside.
On 13.08.2018 21:47, Robert Haas wrote:
> On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan
> <[email protected]> wrote:
>> This thread appears to have gone quiet. What concerns me is that there
>> appears to be substantial disagreement between the author and the reviewers.
>> Since the last thing was this new patch it should really have been put back
>> into "needs review" (my fault to some extent - I missed that). So rather
>> than return the patch with feedfack I'm going to set it to "needs review"
>> and move it to the next CF. However, if we can't arrive at a consensus about
>> the direction during the next CF it should be returned with feedback.
> I agree with the critiques from Robbie Harwood and Michael Paquier
> that the way in that compression is being hooked into the existing
> architecture looks like a kludge. I'm not sure I know exactly how it
> should be done, but the current approach doesn't look natural; it
> looks like it was bolted on.
Sorry, I know that it is too impertinently to ask you or somebody else
to suggest better way of integration compression in libpq
frontend/backend. My primary intention was to use the same code both for
backend and frontend. This is why I pass pointer to receive/send function
to compression module. I am passing secure_read/secure_write function
for backend and pqsecure_read/pqsecure_write functions for frontend.
And change pq_recvbuf/pqReadData, internal_flush/pqSendSome functions to
call zpq_read/zpq_write functions when compression is enabled.
Robbie thinks that compression should be toggled in
secure_write/secure_write/pqsecure_read/pqsecure_write.
I do not understand why it is better then my implementation. From my
point of view it just introduce extra code redundancy and has no advantages.
Just name of this functions (secure_*) will be confusing if this
function will handle compression as well. May be it is better to
introduce some other set of function
which in turn will secuire_* functions, but I also do no think that it
is good idea.
> I agree with the critique from Peter
> Eisentraut and others that we should not go around and add -Z as a
> command-line option to all of our tools; this feature hasn't yet
> proved itself to be useful enough to justify that. Better to let
> people specify it via a connection string option if they want it.
It is not a big deal to remove command line options.
My only concern was that if somebody want to upload data using psql+COPY
command,
it will be more difficult to completely change the way of invoking psql
in this case rather than just adding one option
 (most of user are using -h -D -U options rather than passing complete
connection string).
> I think Thomas Munro was right to ask about what will happen when
> different compression libraries are in use, and I think failing
> uncleanly is quite unacceptable. I think there needs to be some
> method for negotiating the compression type; the client can, for
> example, provide a comma-separated list of methods it supports in
> preference order, and the server can pick the first one it likes.
I will add checking of supported compression method.
Right now Postgres can be configured to use either zlib, either zstd
streaming compression, but not both of them.
So choosing appreciate compression algorithm is not possible, I can only
check that client and server are supporting the same compression method.
Certainly it is possible to implement support of different compression
method and make it possible to chose on at runtime, but I do not think
that it is really good idea
unless we want to support custom compression methods.
> In
> short, I think that a number of people have provided really good
> feedback on this patch, and I suggest to Konstantin that he should
> consider accepting all of those suggestions.
>
> Commit ae65f6066dc3d19a55f4fdcd3b30003c5ad8dbed tried to introduce
> some facilities that can be used for protocol version negotiation as
> new features are added, but this patch doesn't use them. It looks to
> me like it instead just breaks backward compatibility. The new
> 'compression' option won't be recognized by older servers. If it were
> spelled '_pq_.compression' then the facilities in that commit would
> cause a NegotiateProtocolVersion message to be sent by servers which
> have that commit but don't support the compression option. I'm not
> exactly sure what will happen on even-older servers that don't have
> that commit either, but I hope they'll treat it as a GUC name; note
> that setting an unknown GUC name with a namespace prefix is not an
> error, but setting one without such a prefix IS an ERROR. Servers
> which do support compression would respond with a message indicating
> that compression had been enabled or, maybe, just start sending back
> compressed-packet messages, if we go with including some framing in
> the libpq protocol.
>
If I tried to login with new client with _pq_.compression option to old
server (9.6.8) then I got the following error:
knizhnik@knizhnik:~/dtm-data$ psql -d "port=5432 _pq_.compression=zlib
dbname=postgres"
psql: expected authentication request from server, but received v
Frankly speaking I do not see big problem here: the error will happen
only if we use NEW client to connect OLD server and EXPLICITLY specify
compression=on option in connection string.
There will be no problem if we use old client to access new server or
use new client to access old server without switching on compression.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-05 06:04 ` Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 14:10 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 2 replies; 52+ messages in thread
From: Thomas Munro @ 2018-06-05 06:04 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
<[email protected]> wrote:
> Concerning specification of compression level: I have made many experiments
> with different data sets and both zlib/zstd and in both cases using
> compression level higher than default doesn't cause some noticeable increase
> of compression ratio, but quite significantly reduce speed. Moreover, for
> "pgbench -i" zstd provides better compression ratio (63 times!) with
> compression level 1 than with with largest recommended compression level 22!
> This is why I decided not to allow user to choose compression level.
Speaking of configuration, are you planning to support multiple
compression libraries at the same time? It looks like the current
patch implicitly requires client and server to use the same configure
option, without any attempt to detect or negotiate. Do I guess
correctly that a library mismatch would produce an incomprehensible
corrupt stream message?
--
Thomas Munro
http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
@ 2018-06-05 07:09 ` Michael Paquier <[email protected]>
2018-06-05 15:58 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
1 sibling, 2 replies; 52+ messages in thread
From: Michael Paquier @ 2018-06-05 07:09 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote:
> On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
> Speaking of configuration, are you planning to support multiple
> compression libraries at the same time? It looks like the current
> patch implicitly requires client and server to use the same configure
> option, without any attempt to detect or negotiate. Do I guess
> correctly that a library mismatch would produce an incomprehensible
> corrupt stream message?
I just had a quick look at this patch, lured by the smell of your latest
messages... And it seems to me that this patch needs a heavy amount of
work as presented. There are a couple of things which are not really
nice, like forcing the presentation of the compression option in the
startup packet to begin with. The high-jacking around secure_read() is
not nice either as it is aimed at being a rather high-level API on top
of the method used with the backend. On top of adding some
documentation, I think that you could get some inspiration from the
recent GSSAPI encription patch which has been submitted again for v12
cycle, which has spent a large amount of time designing its set of
options.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
@ 2018-06-05 15:58 ` Konstantin Knizhnik <[email protected]>
2018-06-06 07:53 ` Re: libpq compression Michael Paquier <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-05 15:58 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 05.06.2018 10:09, Michael Paquier wrote:
> On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote:
>> On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
>> Speaking of configuration, are you planning to support multiple
>> compression libraries at the same time? It looks like the current
>> patch implicitly requires client and server to use the same configure
>> option, without any attempt to detect or negotiate. Do I guess
>> correctly that a library mismatch would produce an incomprehensible
>> corrupt stream message?
> I just had a quick look at this patch, lured by the smell of your latest
> messages... And it seems to me that this patch needs a heavy amount of
> work as presented. There are a couple of things which are not really
> nice, like forcing the presentation of the compression option in the
> startup packet to begin with. The high-jacking around secure_read() is
> not nice either as it is aimed at being a rather high-level API on top
> of the method used with the backend. On top of adding some
> documentation, I think that you could get some inspiration from the
> recent GSSAPI encription patch which has been submitted again for v12
> cycle, which has spent a large amount of time designing its set of
> options.
> --
> Michael
Thank you for feedback,
I have considered this patch mostly as prototype to estimate efficiency
of libpq protocol compression and compare it with SSL compression.
So I agree with you that there are a lot of things which should be improved.
But can you please clarify what is wrong with "forcing the presentation
of the compression option in the startup packet to begin with"?
Do you mean that it will be better to be able switch on/off compression
during session?
Also I do not completely understand what do you mean by "high-jacking
around secure_read()".
I looked at GSSAPI patch. It does injection in secure_read:
+#ifdef ENABLE_GSS
+Â Â Â if (port->gss->enc)
+Â Â Â {
+Â Â Â Â Â Â n = be_gssapi_read(port, ptr, len);
+Â Â Â Â Â Â waitfor = WL_SOCKET_READABLE;
+Â Â Â }
+Â Â Â else
But the main difference between encryption and compression is that
encryption is not changing data size, while compression does.
To be able to use streaming compression, I need to specify some function
for reading data from the stream. I am using secure_read for this purpose:
     PqStream = zpq_create((zpq_tx_func)secure_write,
(zpq_rx_func)secure_read, MyProcPort);
Can you please explain what is the problem with it?
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 15:58 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-06 07:53 ` Michael Paquier <[email protected]>
2018-06-06 09:30 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Michael Paquier @ 2018-06-06 07:53 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jun 05, 2018 at 06:58:42PM +0300, Konstantin Knizhnik wrote:
> I have considered this patch mostly as prototype to estimate efficiency of
> libpq protocol compression and compare it with SSL compression.
> So I agree with you that there are a lot of things which should be
> improved.
Cool. It seems that there is some meaning for such a feature with
environments with spare CPU and network limitations.
> But can you please clarify what is wrong with "forcing the presentation of
> the compression option in the startup packet to begin with"?
Sure, I am referring to that in your v4:
if (conn->replication && conn->replication[0])
ADD_STARTUP_OPTION("replication", conn->replication);
+ if (conn->compression && conn->compression[0])
+ ADD_STARTUP_OPTION("compression", conn->compression);
There is no point in adding that as a mandatory field of the startup
packet.
> Do you mean that it will be better to be able switch on/off compression
> during session?
Not really, I get that this should be defined when the session is
established and remain until the session finishes. You have a couple of
restrictions like what to do with the first set of messages exchanged
but that could be delayed until the negotiation is done.
> But the main difference between encryption and compression is that
> encryption is not changing data size, while compression does.
> To be able to use streaming compression, I need to specify some function for
> reading data from the stream. I am using secure_read for this purpose:
>
> Â Â Â Â Â PqStream = zpq_create((zpq_tx_func)secure_write,
> (zpq_rx_func)secure_read, MyProcPort);
>
> Can you please explain what is the problem with it?
Likely I have not looked at your patch sufficiently, but the point I am
trying to make is that secure_read or pqsecure_read are entry points
which switch method depending on the connection details. The GSSAPI
encryption patch does that. Yours does not with stuff like that:
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
This makes the whole error handling more consistent, and the new option
layer as well more consistent with what happens in SSL, except that you
want to be able to combine SSL and compression as well so you need an
extra process which decompresses/compresses the data after doing a "raw"
or "ssl" read/write. I have not actually looked much at your patch, but
I am wondering if it could not be possible to make the whole footprint
less invasive which really worries me as now shaped. As you need to
register functions with say zpq_create(), it would be instinctively
nicer to do the handling directly in secure_read() and such.
Just my 2c.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 15:58 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 07:53 ` Re: libpq compression Michael Paquier <[email protected]>
@ 2018-06-06 09:30 ` Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-06 09:30 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Thomas Munro <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 06.06.2018 10:53, Michael Paquier wrote:
> On Tue, Jun 05, 2018 at 06:58:42PM +0300, Konstantin Knizhnik wrote:
>> I have considered this patch mostly as prototype to estimate efficiency of
>> libpq protocol compression and compare it with SSL compression.
>> So I agree with you that there are a lot of things which should be
>> improved.
> Cool. It seems that there is some meaning for such a feature with
> environments with spare CPU and network limitations.
>
>> But can you please clarify what is wrong with "forcing the presentation of
>> the compression option in the startup packet to begin with"?
> Sure, I am referring to that in your v4:
> if (conn->replication && conn->replication[0])
> ADD_STARTUP_OPTION("replication", conn->replication);
> + if (conn->compression && conn->compression[0])
> + ADD_STARTUP_OPTION("compression", conn->compression);
> There is no point in adding that as a mandatory field of the startup
> packet.
Sorry, but ADD_STARTUP_OPTION is not adding manatory field of startup
package. This option any be omitted.
There are a lto of other options registered using ADD_STARTUP_OPTION,
for example all environment-driven GUCs:
   /* Add any environment-driven GUC settings needed */
   for (next_eo = options; next_eo->envName; next_eo++)
   {
      if ((val = getenv(next_eo->envName)) != NULL)
      {
         if (pg_strcasecmp(val, "default") != 0)
            ADD_STARTUP_OPTION(next_eo->pgName, val);
      }
   }
So I do not understand what is wrong here registering "compression" as
option of startup package and what is the alternative for it...
>> Do you mean that it will be better to be able switch on/off compression
>> during session?
> Not really, I get that this should be defined when the session is
> established and remain until the session finishes. You have a couple of
> restrictions like what to do with the first set of messages exchanged
> but that could be delayed until the negotiation is done.
>
>> But the main difference between encryption and compression is that
>> encryption is not changing data size, while compression does.
>> To be able to use streaming compression, I need to specify some function for
>> reading data from the stream. I am using secure_read for this purpose:
>>
>> Â Â Â Â Â PqStream = zpq_create((zpq_tx_func)secure_write,
>> (zpq_rx_func)secure_read, MyProcPort);
>>
>> Can you please explain what is the problem with it?
> Likely I have not looked at your patch sufficiently, but the point I am
> trying to make is that secure_read or pqsecure_read are entry points
> which switch method depending on the connection details. The GSSAPI
> encryption patch does that. Yours does not with stuff like that:
>
> retry4:
> - nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
> - conn->inBufSize - conn->inEnd);
>
> This makes the whole error handling more consistent, and the new option
> layer as well more consistent with what happens in SSL, except that you
> want to be able to combine SSL and compression as well so you need an
> extra process which decompresses/compresses the data after doing a "raw"
> or "ssl" read/write. I have not actually looked much at your patch, but
> I am wondering if it could not be possible to make the whole footprint
> less invasive which really worries me as now shaped. As you need to
> register functions with say zpq_create(), it would be instinctively
> nicer to do the handling directly in secure_read() and such.
Once again sorry, but i still do not understand the problem here.
If compression is anabled, then I am using zpq_read instead of
secure_read/pqsecure_read. But zpq_read is in turn calling
secure_read/pqsecure_read
to fetch more raw data. So if "ecure_read or pqsecure_read are entry
points which switch method depending on the connection details", thenÂ
compression is not preventing them from making this choice. Compression
should be done prior to encryption otherwise compression will have no sense.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
@ 2018-06-05 17:06 ` Peter Eisentraut <[email protected]>
2018-06-05 19:58 ` Re: libpq compression Dave Cramer <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 2 replies; 52+ messages in thread
From: Peter Eisentraut @ 2018-06-05 17:06 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 6/5/18 03:09, Michael Paquier wrote:
> I just had a quick look at this patch, lured by the smell of your latest
> messages... And it seems to me that this patch needs a heavy amount of
> work as presented. There are a couple of things which are not really
> nice, like forcing the presentation of the compression option in the
> startup packet to begin with.
Yeah, at this point we will probably need a discussion and explanation
of the protocol behavior this is adding, such as how to negotiate
different compression settings.
Unrelatedly, I suggest skipping the addition of -Z options to various
client-side tools. This is unnecessary, since generic connection
options can already be specified via -d typically, and it creates
confusion because -Z is already used to specify output compression by
some programs.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
@ 2018-06-05 19:58 ` Dave Cramer <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Dave Cramer @ 2018-06-05 19:58 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; Konstantin Knizhnik <[email protected]>; Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 5 June 2018 at 13:06, Peter Eisentraut <[email protected]>
wrote:
> On 6/5/18 03:09, Michael Paquier wrote:
> > I just had a quick look at this patch, lured by the smell of your latest
> > messages... And it seems to me that this patch needs a heavy amount of
> > work as presented. There are a couple of things which are not really
> > nice, like forcing the presentation of the compression option in the
> > startup packet to begin with.
>
> Yeah, at this point we will probably need a discussion and explanation
> of the protocol behavior this is adding, such as how to negotiate
> different compression settings.
>
> Unrelatedly, I suggest skipping the addition of -Z options to various
> client-side tools. This is unnecessary, since generic connection
> options can already be specified via -d typically, and it creates
> confusion because -Z is already used to specify output compression by
> some programs.
>
>
As the maintainer of the JDBC driver I would think we would also like to
leverage this as well.
There are a few other drivers that implement the protocol as well and I'm
sure they would want in as well.
I haven't looked at the patch but if we get to the point of negotiating
compression please let me know.
Thanks,
Dave Cramer
[email protected]
www.postgresintl.com
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
@ 2018-06-06 16:33 ` Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-06 16:33 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; Michael Paquier <[email protected]>; Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 05.06.2018 20:06, Peter Eisentraut wrote:
> On 6/5/18 03:09, Michael Paquier wrote:
>> I just had a quick look at this patch, lured by the smell of your latest
>> messages... And it seems to me that this patch needs a heavy amount of
>> work as presented. There are a couple of things which are not really
>> nice, like forcing the presentation of the compression option in the
>> startup packet to begin with.
> Yeah, at this point we will probably need a discussion and explanation
> of the protocol behavior this is adding, such as how to negotiate
> different compression settings.
>
> Unrelatedly, I suggest skipping the addition of -Z options to various
> client-side tools. This is unnecessary, since generic connection
> options can already be specified via -d typically, and it creates
> confusion because -Z is already used to specify output compression by
> some programs.
>
Sorry, psql is using '-d' option for specifying database name and
pgbench is using '-d' option for toggling debug output.
So may be there is some other way to pass generic connection option, but
in any case it seems to be less convenient for users.
Also I do not see any contradiction with using -Z option in some other
tools (pg_basebackup, pg_receivewal, pg_dump)
for enabling output compression. It will be bad if that option has
contradictory meaning in different tools. But if it is used for toggling
compression
(doesn't matter at which level), then I do not see that it can be source
of confusion.
The only problem is with pg_dump which establish connection with server
to fetch data from the database and is able to compress output data.
So here we may need two options: compress input and compress output.Â
But I do not think that because of it -Z option should be removed from
psql and pgbench.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-06 17:20 ` Konstantin Knizhnik <[email protected]>
2018-06-06 17:22 ` Re: libpq compression Joshua D. Drake <[email protected]>
2018-06-06 20:01 ` Re: libpq compression Peter Eisentraut <[email protected]>
0 siblings, 2 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-06 17:20 UTC (permalink / raw)
To: [email protected]
On 06.06.2018 19:33, Konstantin Knizhnik wrote:
>
>
> On 05.06.2018 20:06, Peter Eisentraut wrote:
>> On 6/5/18 03:09, Michael Paquier wrote:
>>> I just had a quick look at this patch, lured by the smell of your
>>> latest
>>> messages... And it seems to me that this patch needs a heavy amount of
>>> work as presented. There are a couple of things which are not really
>>> nice, like forcing the presentation of the compression option in the
>>> startup packet to begin with.
>> Yeah, at this point we will probably need a discussion and explanation
>> of the protocol behavior this is adding, such as how to negotiate
>> different compression settings.
>>
>> Unrelatedly, I suggest skipping the addition of -Z options to various
>> client-side tools. This is unnecessary, since generic connection
>> options can already be specified via -d typically, and it creates
>> confusion because -Z is already used to specify output compression by
>> some programs.
>>
>
> Sorry, psql is using '-d' option for specifying database name and
> pgbench is using '-d' option for toggling debug output.
> So may be there is some other way to pass generic connection option,
> but in any case it seems to be less convenient for users.
> Also I do not see any contradiction with using -Z option in some other
> tools (pg_basebackup, pg_receivewal, pg_dump)
> for enabling output compression. It will be bad if that option has
> contradictory meaning in different tools. But if it is used for
> toggling compression
> (doesn't matter at which level), then I do not see that it can be
> source of confusion.
>
> The only problem is with pg_dump which establish connection with
> server to fetch data from the database and is able to compress output
> data.
> So here we may need two options: compress input and compress output.Â
> But I do not think that because of it -Z option should be removed from
> psql and pgbench.
>
>
Well, psql really allows to specify complete connection string with -d
options (although it is not mentioned in help).
But still I think that it is inconvenient to require user to write
complete connection string to be able to specify compression option,
while everybody prefer to use -h, -U, -p options to specify
correspondent components of connection string.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-06 17:22 ` Joshua D. Drake <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Joshua D. Drake @ 2018-06-06 17:22 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; [email protected]
On 06/06/2018 10:20 AM, Konstantin Knizhnik wrote:
> Well, psql really allows to specify complete connection string with -d
> options (although it is not mentioned in help).
> But still I think that it is inconvenient to require user to write
> complete connection string to be able to specify compression option,
> while everybody prefer to use -h, -U, -p options to specify
> correspondent components of connection string.
From a barrier to entry and simplicity sake I agree. We should have a
standard flag.
JD
--
Command Prompt, Inc. || http://the.postgres.company/ || @cmdpromptinc
*** A fault and talent of mine is to tell it exactly how it is. ***
PostgreSQL centered full stack support, consulting and development.
Advocate: @amplifypostgres || Learn: https://postgresconf.org
***** Unless otherwise stated, opinions are my own. *****
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
@ 2018-06-06 20:01 ` Peter Eisentraut <[email protected]>
2018-06-06 20:16 ` libpq compression David G. Johnston <[email protected]>
2018-06-07 02:22 ` Re: libpq compression Craig Ringer <[email protected]>
1 sibling, 2 replies; 52+ messages in thread
From: Peter Eisentraut @ 2018-06-06 20:01 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; [email protected]
On 6/6/18 13:20, Konstantin Knizhnik wrote:
> Well, psql really allows to specify complete connection string with -d
> options (although it is not mentioned in help).
> But still I think that it is inconvenient to require user to write
> complete connection string to be able to specify compression option,
> while everybody prefer to use -h, -U, -p options to specify
> correspondent components of connection string.
I recommend that you avoid derailing your effort by hinging it on this
issue. You can always add command-line options after the libpq support
is in.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 52+ messages in thread
* libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 20:01 ` Re: libpq compression Peter Eisentraut <[email protected]>
@ 2018-06-06 20:16 ` David G. Johnston <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: David G. Johnston @ 2018-06-06 20:16 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; [email protected] <[email protected]>
On Wednesday, June 6, 2018, Peter Eisentraut <peter.eisentraut@2ndquadrant.
com> wrote:
> On 6/6/18 13:20, Konstantin Knizhnik wrote:
> > Well, psql really allows to specify complete connection string with -d
> > options (although it is not mentioned in help).
> > But still I think that it is inconvenient to require user to write
> > complete connection string to be able to specify compression option,
> > while everybody prefer to use -h, -U, -p options to specify
> > correspondent components of connection string.
>
> I recommend that you avoid derailing your effort by hinging it on this
> issue. You can always add command-line options after the libpq support
> is in.
>
>
It probably requires a long option. It can be debated whether a short
option is warranted (as well as an environment variable).
David J.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
2018-06-05 07:09 ` Re: libpq compression Michael Paquier <[email protected]>
2018-06-05 17:06 ` Re: libpq compression Peter Eisentraut <[email protected]>
2018-06-06 16:33 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-06 20:01 ` Re: libpq compression Peter Eisentraut <[email protected]>
@ 2018-06-07 02:22 ` Craig Ringer <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Craig Ringer @ 2018-06-07 02:22 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; PostgreSQL Hackers <[email protected]>
On 7 June 2018 at 04:01, Peter Eisentraut <[email protected]>
wrote:
> On 6/6/18 13:20, Konstantin Knizhnik wrote:
> > Well, psql really allows to specify complete connection string with -d
> > options (although it is not mentioned in help).
> > But still I think that it is inconvenient to require user to write
> > complete connection string to be able to specify compression option,
> > while everybody prefer to use -h, -U, -p options to specify
> > correspondent components of connection string.
>
> I recommend that you avoid derailing your effort by hinging it on this
> issue. You can always add command-line options after the libpq support
> is in.
>
Strongly agree. Let libpq handle it first with the core protocol support
and connstr parsing, add convenience flags later.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: libpq compression
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Re: libpq compression Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Re: libpq compression Thomas Munro <[email protected]>
@ 2018-06-05 14:10 ` Konstantin Knizhnik <[email protected]>
1 sibling, 0 replies; 52+ messages in thread
From: Konstantin Knizhnik @ 2018-06-05 14:10 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Grigory Smolkin <[email protected]>; PostgreSQL Hackers <[email protected]>
On 05.06.2018 09:04, Thomas Munro wrote:
> On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik
> <[email protected]> wrote:
>> Concerning specification of compression level: I have made many experiments
>> with different data sets and both zlib/zstd and in both cases using
>> compression level higher than default doesn't cause some noticeable increase
>> of compression ratio, but quite significantly reduce speed. Moreover, for
>> "pgbench -i" zstd provides better compression ratio (63 times!) with
>> compression level 1 than with with largest recommended compression level 22!
>> This is why I decided not to allow user to choose compression level.
> Speaking of configuration, are you planning to support multiple
> compression libraries at the same time? It looks like the current
> patch implicitly requires client and server to use the same configure
> option, without any attempt to detect or negotiate. Do I guess
> correctly that a library mismatch would produce an incomprehensible
> corrupt stream message?
>
Frankly speaking I am not sure that support of multiple compression
libraries at the same time is actually needed.
If we build Postgres from sources, then both frontend and backend
libraries will use the same compression library.
zlib is available almost everywhere and Postgres in any case is using it.
zstd is faster and provides better compression ratio. So in principle it
may be useful to try first to use zstd and if it is not available then
use zlib.
It will require dynamic loading of this libraries.
libpq stream compression is not the only place where compression is used
in Postgres. So I think that the problem of choosing compression algorithm
and supporting custom compression methods should be fixed at some upper
level. There is patch for custom compression method at commit fest.
May be it should be combined with this one.
Right now if client and server libpq libraries were built with different
compression libraries, then decompress error will be reported.
Supporting multiple compression methods will require more sophisticated
handshake protocol so that client and server can choose compression
method which is supported by both of them.
But certainly it can be done.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v3 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 26 +++---
contrib/dblink/expected/dblink.out | 1 -
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 23 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 3 +-
contrib/postgres_fdw/option.c | 22 +++--
src/backend/foreign/foreign.c | 25 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 6 +-
10 files changed, 155 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index e323fdd0e6..5e2d7b4c70 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,31 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..14d015e4d5 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,6 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..53d996bd7c 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,31 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..2588ad4aa6 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,6 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9626,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Did you mean "passfile"?
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..6971c88463 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,30 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..283ebb6a58 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,31 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 8539cef024..174f9c390d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6196,6 +6196,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance exceeds the maximum
+ * allowed or more than half the characters are different, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ dist <= strlen(state->source) / 2 &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..97a0b9cda1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,6 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +439,6 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +595,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +634,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--gBBFr7Ir9EOA20Yy--
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v2 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 26 +++---
contrib/dblink/expected/dblink.out | 2 +-
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 23 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 4 +-
contrib/postgres_fdw/option.c | 22 +++--
src/backend/foreign/foreign.c | 25 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 8 +-
10 files changed, 159 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index e323fdd0e6..c134f73adb 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,31 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..2b874fc450 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,7 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..e3b69ec125 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,31 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..036eaa63df 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,7 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
+HINT: Did you mean "sslcert"?
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9627,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Did you mean "passfile"?
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..5ac0484c3e 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,30 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..0e2004ec8c 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,31 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 8539cef024..55905c5f5d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6196,6 +6196,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance between the source
+ * string and the candidate string exceeds the maximum distance allowed by the
+ * state, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..588ce62266 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,7 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
+HINT: Did you mean "host"?
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +440,7 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
+HINT: Did you mean "host"?
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +597,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +636,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--VbJkn9YxBvnuCH5J--
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v2 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 26 +++---
contrib/dblink/expected/dblink.out | 2 +-
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 23 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 4 +-
contrib/postgres_fdw/option.c | 22 +++--
src/backend/foreign/foreign.c | 25 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 8 +-
10 files changed, 159 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index e323fdd0e6..c134f73adb 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,31 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..2b874fc450 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,7 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..e3b69ec125 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,31 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..036eaa63df 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,7 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
+HINT: Did you mean "sslcert"?
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9627,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Did you mean "passfile"?
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..5ac0484c3e 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,30 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..0e2004ec8c 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,31 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 5);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 8539cef024..55905c5f5d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6196,6 +6196,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance between the source
+ * string and the candidate string exceeds the maximum distance allowed by the
+ * state, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..588ce62266 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,7 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
+HINT: Did you mean "host"?
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +440,7 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
+HINT: Did you mean "host"?
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +597,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +636,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--VbJkn9YxBvnuCH5J--
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v4 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 27 +++---
contrib/dblink/expected/dblink.out | 1 -
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 24 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 3 +-
contrib/postgres_fdw/option.c | 23 ++++--
src/backend/foreign/foreign.c | 26 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 6 +-
10 files changed, 159 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index 3df3f9bbe9..8c26adb56c 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,32 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..14d015e4d5 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,6 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..306d349656 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,32 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..ddc185b7d0 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,6 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9626,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Perhaps you meant to reference the option "passfile".
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..b1a14d204b 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,31 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..4ef4a765f4 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,32 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 816c66b7e7..1f6e090821 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6197,6 +6197,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance exceeds the maximum
+ * allowed or more than half the characters are different, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ dist <= strlen(state->source) / 2 &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..1b4c19a49b 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,6 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +439,6 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +595,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Perhaps you meant to reference the option "user".
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +634,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Perhaps you meant to reference the option "user".
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--Nq2Wo0NMKNjxTN9z--
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v3 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 26 +++---
contrib/dblink/expected/dblink.out | 1 -
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 23 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 3 +-
contrib/postgres_fdw/option.c | 22 +++--
src/backend/foreign/foreign.c | 25 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 6 +-
10 files changed, 155 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index e323fdd0e6..5e2d7b4c70 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,31 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..14d015e4d5 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,6 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..53d996bd7c 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,31 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..2588ad4aa6 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,6 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9626,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Did you mean "passfile"?
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..6971c88463 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,30 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..283ebb6a58 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,31 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Did you mean \"%s\"?", closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 8539cef024..174f9c390d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6196,6 +6196,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance exceeds the maximum
+ * allowed or more than half the characters are different, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ dist <= strlen(state->source) / 2 &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..97a0b9cda1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,6 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +439,6 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +595,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +634,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Did you mean "user"?
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--gBBFr7Ir9EOA20Yy--
^ permalink raw reply [nested|flat] 52+ messages in thread
* [PATCH v4 1/1] Adjust assorted hint messages that list all valid options.
@ 2022-09-02 21:03 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Nathan Bossart @ 2022-09-02 21:03 UTC (permalink / raw)
Instead of listing all valid options, we now try to provide one
that looks similar. Since this may be useful elsewhere, this
change introduces a new set of functions that can be reused for
similar purposes.
---
contrib/dblink/dblink.c | 27 +++---
contrib/dblink/expected/dblink.out | 1 -
contrib/file_fdw/expected/file_fdw.out | 2 -
contrib/file_fdw/file_fdw.c | 24 ++++--
.../postgres_fdw/expected/postgres_fdw.out | 3 +-
contrib/postgres_fdw/option.c | 23 ++++--
src/backend/foreign/foreign.c | 26 ++++--
src/backend/utils/adt/varlena.c | 82 +++++++++++++++++++
src/include/utils/varlena.h | 12 +++
src/test/regress/expected/foreign_data.out | 6 +-
10 files changed, 159 insertions(+), 47 deletions(-)
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index 3df3f9bbe9..8c26adb56c 100644
--- a/contrib/dblink/dblink.c
+++ b/contrib/dblink/dblink.c
@@ -2008,27 +2008,32 @@ dblink_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option, or invalid option for the context specified, so
- * complain about it. Provide a hint with list of valid options
- * for the context.
+ * complain about it. Provide a hint with a valid option that
+ * looks similar, if there is one.
*/
- StringInfoData buf;
const PQconninfoOption *opt;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = options; opt->keyword; opt++)
{
if (is_valid_dblink_option(options, opt->keyword, context))
- appendStringInfo(&buf, "%s%s",
- (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_OPTION_NAME_NOT_FOUND),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
}
diff --git a/contrib/dblink/expected/dblink.out b/contrib/dblink/expected/dblink.out
index c7bde6ad07..14d015e4d5 100644
--- a/contrib/dblink/expected/dblink.out
+++ b/contrib/dblink/expected/dblink.out
@@ -897,7 +897,6 @@ $d$;
CREATE USER MAPPING FOR public SERVER fdtest
OPTIONS (server 'localhost'); -- fail, can't specify server here
ERROR: invalid option "server"
-HINT: Valid options in this context are: user, password, sslpassword
CREATE USER MAPPING FOR public SERVER fdtest OPTIONS (user :'USER');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT EXECUTE ON FUNCTION dblink_connect_u(text, text) TO regress_dblink_user;
diff --git a/contrib/file_fdw/expected/file_fdw.out b/contrib/file_fdw/expected/file_fdw.out
index 261af1a8b5..36d76ba26c 100644
--- a/contrib/file_fdw/expected/file_fdw.out
+++ b/contrib/file_fdw/expected/file_fdw.out
@@ -166,7 +166,6 @@ ERROR: invalid option "force_not_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_not_null '*'); -- ERROR
ERROR: invalid option "force_not_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- force_null is not allowed to be specified at any foreign object level:
ALTER FOREIGN DATA WRAPPER file_fdw OPTIONS (ADD force_null '*'); -- ERROR
ERROR: invalid option "force_null"
@@ -179,7 +178,6 @@ ERROR: invalid option "force_null"
HINT: There are no valid options in this context.
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (force_null '*'); -- ERROR
ERROR: invalid option "force_null"
-HINT: Valid options in this context are: filename, program, format, header, delimiter, quote, escape, null, encoding
-- basic query tests
SELECT * FROM agg_text WHERE b > 10.0 ORDER BY a;
a | b
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 4773cadec0..306d349656 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -37,6 +37,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/sampling.h"
+#include "utils/varlena.h"
PG_MODULE_MAGIC;
@@ -214,27 +215,32 @@ file_fdw_validator(PG_FUNCTION_ARGS)
if (!is_valid_option(def->defname, catalog))
{
const struct FileFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = valid_options; opt->optname; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7bf35602b0..ddc185b7d0 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -188,7 +188,6 @@ ALTER USER MAPPING FOR public SERVER testserver1
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslmode 'require');
ERROR: invalid option "sslmode"
-HINT: Valid options in this context are: user, password, sslpassword, password_required, sslcert, sslkey
-- But we can add valid ones fine
ALTER USER MAPPING FOR public SERVER testserver1
OPTIONS (ADD sslpassword 'dummy');
@@ -9627,7 +9626,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, parallel_commit, keep_connections
+HINT: Perhaps you meant to reference the option "passfile".
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 95dde056eb..b1a14d204b 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -90,26 +90,31 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
{
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
PgFdwOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = postgres_fdw_options; opt->keyword; opt++)
{
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->keyword);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->keyword);
+ }
}
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_FDW_INVALID_OPTION_NAME),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
}
/*
diff --git a/src/backend/foreign/foreign.c b/src/backend/foreign/foreign.c
index cf222fc3e9..4ef4a765f4 100644
--- a/src/backend/foreign/foreign.c
+++ b/src/backend/foreign/foreign.c
@@ -27,6 +27,7 @@
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/syscache.h"
+#include "utils/varlena.h"
/*
@@ -621,25 +622,32 @@ postgresql_fdw_validator(PG_FUNCTION_ARGS)
if (!is_conninfo_option(def->defname, catalog))
{
const struct ConnectionOption *opt;
- StringInfoData buf;
+ const char *closest_match;
+ ClosestMatchState match_state;
+ bool has_valid_options = false;
/*
* Unknown option specified, complain about it. Provide a hint
- * with list of valid options for the object.
+ * with a valid option that looks similar, if there is one.
*/
- initStringInfo(&buf);
+ initClosestMatch(&match_state, def->defname, 4);
for (opt = libpq_conninfo_options; opt->optname; opt++)
+ {
if (catalog == opt->optcontext)
- appendStringInfo(&buf, "%s%s", (buf.len > 0) ? ", " : "",
- opt->optname);
+ {
+ has_valid_options = true;
+ updateClosestMatch(&match_state, opt->optname);
+ }
+ }
+ closest_match = getClosestMatch(&match_state);
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("invalid option \"%s\"", def->defname),
- buf.len > 0
- ? errhint("Valid options in this context are: %s",
- buf.data)
- : errhint("There are no valid options in this context.")));
+ has_valid_options ? closest_match ?
+ errhint("Perhaps you meant to reference the option \"%s\".",
+ closest_match) : 0 :
+ errhint("There are no valid options in this context.")));
PG_RETURN_BOOL(false);
}
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 816c66b7e7..1f6e090821 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -6197,6 +6197,88 @@ rest_of_char_same(const char *s1, const char *s2, int len)
#include "levenshtein.c"
+/*
+ * The following *ClosestMatch() functions can be used to determine whether a
+ * user-provided string resembles any known valid values, which is useful for
+ * providing hints in log messages, among other things. Use these functions
+ * like so:
+ *
+ * initClosestMatch(&state, source_string, max_distance);
+ *
+ * for (int i = 0; i < num_valid_strings; i++)
+ * updateClosestMatch(&state, valid_strings[i]);
+ *
+ * closestMatch = getClosestMatch(&state);
+ */
+
+/*
+ * Initialize the given state with the source string and maximum Levenshtein
+ * distance to consider.
+ */
+void
+initClosestMatch(ClosestMatchState *state, const char *source, int max_d)
+{
+ Assert(state);
+ Assert(max_d >= 0);
+
+ state->source = source;
+ state->min_d = -1;
+ state->max_d = max_d;
+ state->match = NULL;
+}
+
+/*
+ * If the candidate string is a closer match than the current one saved (or
+ * there is no match saved), save it as the closest match.
+ *
+ * If the source or candidate string is NULL, empty, or too long, this function
+ * takes no action. Likewise, if the Levenshtein distance exceeds the maximum
+ * allowed or more than half the characters are different, no action is taken.
+ */
+void
+updateClosestMatch(ClosestMatchState *state, const char *candidate)
+{
+ int dist;
+
+ Assert(state);
+
+ if (state->source == NULL || state->source[0] == '\0' ||
+ candidate == NULL || candidate[0] == '\0')
+ return;
+
+ /*
+ * To avoid ERROR-ing, we check the lengths here instead of setting
+ * 'trusted' to false in the call to varstr_levenshtein_less_equal().
+ */
+ if (strlen(state->source) > MAX_LEVENSHTEIN_STRLEN ||
+ strlen(candidate) > MAX_LEVENSHTEIN_STRLEN)
+ return;
+
+ dist = varstr_levenshtein_less_equal(state->source, strlen(state->source),
+ candidate, strlen(candidate), 1, 1, 1,
+ state->max_d, true);
+ if (dist <= state->max_d &&
+ dist <= strlen(state->source) / 2 &&
+ (state->min_d == -1 || dist < state->min_d))
+ {
+ state->min_d = dist;
+ state->match = candidate;
+ }
+}
+
+/*
+ * Return the closest match. If no suitable candidates were provided via
+ * updateClosestMatch(), return NULL.
+ */
+const char *
+getClosestMatch(ClosestMatchState *state)
+{
+ Assert(state);
+
+ return state->match;
+}
+
+
/*
* Unicode support
*/
diff --git a/src/include/utils/varlena.h b/src/include/utils/varlena.h
index c45208a204..2bc3b6e519 100644
--- a/src/include/utils/varlena.h
+++ b/src/include/utils/varlena.h
@@ -38,4 +38,16 @@ extern text *replace_text_regexp(text *src_text, text *pattern_text,
int cflags, Oid collation,
int search_start, int n);
+typedef struct ClosestMatchState
+{
+ const char *source;
+ int min_d;
+ int max_d;
+ const char *match;
+} ClosestMatchState;
+
+extern void initClosestMatch(ClosestMatchState *state, const char *source, int max_d);
+extern void updateClosestMatch(ClosestMatchState *state, const char *candidate);
+extern const char *getClosestMatch(ClosestMatchState *state);
+
#endif
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 33505352cc..1b4c19a49b 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -329,7 +329,6 @@ CREATE SERVER s6 VERSION '16.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbna
CREATE SERVER s7 TYPE 'oracle' VERSION '17.0' FOREIGN DATA WRAPPER foo OPTIONS (host 'a', dbname 'b');
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (foo '1'); -- ERROR
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
CREATE SERVER s8 FOREIGN DATA WRAPPER postgresql OPTIONS (host 'localhost', dbname 's8db');
\des+
List of foreign servers
@@ -440,7 +439,6 @@ ERROR: permission denied for foreign-data wrapper foo
RESET ROLE;
ALTER SERVER s8 OPTIONS (foo '1'); -- ERROR option validation
ERROR: invalid option "foo"
-HINT: Valid options in this context are: authtype, service, connect_timeout, dbname, host, hostaddr, port, tty, options, requiressl, sslmode, gsslib
ALTER SERVER s8 OPTIONS (connect_timeout '30', SET dbname 'db1', DROP host);
SET ROLE regress_test_role;
ALTER SERVER s1 OWNER TO regress_test_indirect; -- ERROR
@@ -597,7 +595,7 @@ ERROR: user mapping for "regress_foreign_data_user" already exists for server "
CREATE USER MAPPING FOR public SERVER s4 OPTIONS ("this mapping" 'is public');
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (username 'test', password 'secret'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Perhaps you meant to reference the option "user".
CREATE USER MAPPING FOR user SERVER s8 OPTIONS (user 'test', password 'secret');
ALTER SERVER s5 OWNER TO regress_test_role;
ALTER SERVER s6 OWNER TO regress_test_indirect;
@@ -636,7 +634,7 @@ ALTER USER MAPPING FOR public SERVER s5 OPTIONS (gotcha 'true'); -- E
ERROR: user mapping for "public" does not exist for server "s5"
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (username 'test'); -- ERROR
ERROR: invalid option "username"
-HINT: Valid options in this context are: user, password
+HINT: Perhaps you meant to reference the option "user".
ALTER USER MAPPING FOR current_user SERVER s8 OPTIONS (DROP user, SET password 'public');
SET ROLE regress_test_role;
ALTER USER MAPPING FOR current_user SERVER s5 OPTIONS (ADD modified '1');
--
2.25.1
--Nq2Wo0NMKNjxTN9z--
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: File descriptors in exec'd subprocesses
@ 2023-02-05 00:03 Thomas Munro <[email protected]>
2023-02-05 05:15 ` Re: File descriptors in exec'd subprocesses Tom Lane <[email protected]>
0 siblings, 1 reply; 52+ messages in thread
From: Thomas Munro @ 2023-02-05 00:03 UTC (permalink / raw)
To: pgsql-hackers
On Sun, Feb 5, 2023 at 1:00 PM Thomas Munro <[email protected]> wrote:
> SUSv3 (POSIX 2008)
Oh, oops, 2008 actually corresponds to SUSv4. Hmm.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: File descriptors in exec'd subprocesses
2023-02-05 00:03 Re: File descriptors in exec'd subprocesses Thomas Munro <[email protected]>
@ 2023-02-05 05:15 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 52+ messages in thread
From: Tom Lane @ 2023-02-05 05:15 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
Thomas Munro <[email protected]> writes:
> On Sun, Feb 5, 2023 at 1:00 PM Thomas Munro <[email protected]> wrote:
>> SUSv3 (POSIX 2008)
> Oh, oops, 2008 actually corresponds to SUSv4. Hmm.
Worst case, if we come across some allegedly-supported platform without
O_CLOEXEC, we #define that to zero. Said platform is no worse off
than it was before.
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
end of thread, other threads:[~2023-02-05 05:15 UTC | newest]
Thread overview: 52+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-05-16 15:09 Re: libpq compression Grigory Smolkin <[email protected]>
2018-05-16 15:54 ` Konstantin Knizhnik <[email protected]>
2018-06-05 05:26 ` Thomas Munro <[email protected]>
2018-06-05 14:06 ` Konstantin Knizhnik <[email protected]>
2018-06-05 23:03 ` Thomas Munro <[email protected]>
2018-06-06 09:37 ` Konstantin Knizhnik <[email protected]>
2018-06-18 20:34 ` Robbie Harwood <[email protected]>
2018-06-19 07:54 ` Konstantin Knizhnik <[email protected]>
2018-06-19 21:04 ` Robbie Harwood <[email protected]>
2018-06-20 06:40 ` Konstantin Knizhnik <[email protected]>
2018-06-20 20:34 ` Robbie Harwood <[email protected]>
2018-06-21 07:12 ` Konstantin Knizhnik <[email protected]>
2018-06-21 14:56 ` Robbie Harwood <[email protected]>
2018-06-21 16:04 ` Konstantin Knizhnik <[email protected]>
2018-06-21 17:14 ` Robbie Harwood <[email protected]>
2018-06-22 07:11 ` Konstantin Knizhnik <[email protected]>
2018-06-22 15:59 ` Robbie Harwood <[email protected]>
2018-06-22 16:32 ` Konstantin Knizhnik <[email protected]>
2018-06-22 17:56 ` Robbie Harwood <[email protected]>
2018-06-23 11:40 ` Konstantin Knizhnik <[email protected]>
2018-06-21 21:34 ` Nico Williams <[email protected]>
2018-06-22 07:18 ` Konstantin Knizhnik <[email protected]>
2018-06-22 16:05 ` Nico Williams <[email protected]>
2018-06-22 16:35 ` Konstantin Knizhnik <[email protected]>
2018-06-25 09:32 ` Konstantin Knizhnik <[email protected]>
2018-08-10 21:55 ` Andrew Dunstan <[email protected]>
2018-08-13 18:47 ` Robert Haas <[email protected]>
2018-08-13 20:06 ` Andrew Dunstan <[email protected]>
2018-08-20 15:00 ` Konstantin Knizhnik <[email protected]>
2018-08-14 14:50 ` Konstantin Knizhnik <[email protected]>
2018-06-05 06:04 ` Thomas Munro <[email protected]>
2018-06-05 07:09 ` Michael Paquier <[email protected]>
2018-06-05 15:58 ` Konstantin Knizhnik <[email protected]>
2018-06-06 07:53 ` Michael Paquier <[email protected]>
2018-06-06 09:30 ` Konstantin Knizhnik <[email protected]>
2018-06-05 17:06 ` Peter Eisentraut <[email protected]>
2018-06-05 19:58 ` Dave Cramer <[email protected]>
2018-06-06 16:33 ` Konstantin Knizhnik <[email protected]>
2018-06-06 17:20 ` Konstantin Knizhnik <[email protected]>
2018-06-06 17:22 ` Joshua D. Drake <[email protected]>
2018-06-06 20:01 ` Peter Eisentraut <[email protected]>
2018-06-06 20:16 ` David G. Johnston <[email protected]>
2018-06-07 02:22 ` Craig Ringer <[email protected]>
2018-06-05 14:10 ` Konstantin Knizhnik <[email protected]>
2022-09-02 21:03 [PATCH v3 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2022-09-02 21:03 [PATCH v2 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2022-09-02 21:03 [PATCH v2 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2022-09-02 21:03 [PATCH v4 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2022-09-02 21:03 [PATCH v3 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2022-09-02 21:03 [PATCH v4 1/1] Adjust assorted hint messages that list all valid options. Nathan Bossart <[email protected]>
2023-02-05 00:03 Re: File descriptors in exec'd subprocesses Thomas Munro <[email protected]>
2023-02-05 05:15 ` Re: File descriptors in exec'd subprocesses Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox