pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedlibpq compression (part 2)
27+ messages / 10 participants
[nested] [flat]
* libpq compression (part 2)
@ 2021-12-31 09:26 Daniil Zakhlystov <usernamedt@yandex-team.ru>
0 siblings, 2 replies; 27+ messages in thread
From: Daniil Zakhlystov @ 2021-12-31 09:26 UTC (permalink / raw)
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Hi!
I want to introduce the updated version of the libpq protocol compression patch, initially
introduced by Konstantin Knizhnik in this thread:
https://www.postgresql.org/message-id/aad16e41-b3f9-e89d-fa57-fb4c694bec25@postgrespro.ru
The original thread became huge and it makes it hard for new people to catch up so I think it is better
to open a new thread with the summary of the current state of the patch.
Compression of libpq traffic is useful in:
1. COPY
2. Replication
3. Queries returning large results sets (for example JSON) through slow connections.
The patch introduces three new protocol messages: CompressionAck, CompressedData, and
SetCompressionMethod.
Here is a brief overview of the compression initialization process:
1. Compression can be requested by a client by including the "compression" option in its connection
string. This can either be a boolean value to enable or
disable compression or an explicit list of comma-separated compression algorithms which can
optionally include compression level. The client indicates the compression request by sending the
_pq_.compression startup packet
parameter with a list of compression algorithms and an optional specification of compression level.
If the server does not support compression, the backend will ignore the _pq_.compression parameter
and will not send the CompressionAck message to the frontend.
2. Server receives the client's compression request and intersects the requested compression
algorithms with the allowed ones (controlled via the libpq_compression server config setting). If
the intersection is not empty, the server responds with CompressionAck containing the final list of
the compression algorithms that can be used for the compression of libpq messages between the client
and server. If the intersection is empty (server does not accept any of the requested algorithms),
then it replies with CompressionAck containing the empty list and it is up to the client whether to
continue without compression or to report an error.
3. After sending the CompressionAck message, the server can send the SetCompressionMethod message to
set the current compression algorithm for server-to-client traffic compression. Same for the client,
after receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current
compression algorithm for client-to-server traffic compression. Client-to-server and
server-to-client compression are independent of each other.
To compress messages, streaming compression is used. Compressed bytes are wrapped into the
CompressedData protocol messages. One CompressedData message may contain multiple regular protocol
messages. CompressedData messages can be mixed with the regular uncompressed messages.
Compression context is retained between the multiple CompressedData messages. Currently, only
CopyData, DataRow, and Query types of messages with length more than 60 bytes are being compressed.
If the client (or server) wants to switch the current compression method, it sends the
SetCompressionMethod message to the receiving side would be able to change its decompressor.
I've separated the patch into two parts: first contains the main patch with ZLIB and LZ4 compressing
algorithms support, second adds the ZSTD support.
Thanks,
Daniil Zakhlystov
Attachments: download all (2 files, .tar.gz)
0001-Implement-libpq-compression.patch (application/octet-stream, 114.4K) download
From 5ee434ac350215907ef7d6aff3665d362538bcae Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:58:38 +0500
Subject: [PATCH 1/2] Implement libpq compression
---
.../postgres_fdw/expected/postgres_fdw.out | 2 +-
doc/src/sgml/config.sgml | 21 +
doc/src/sgml/libpq.sgml | 37 +
doc/src/sgml/protocol.sgml | 174 +++
src/Makefile.global.in | 1 +
src/backend/Makefile | 8 +
src/backend/catalog/system_views.sql | 9 +
src/backend/libpq/pqcomm.c | 277 ++++-
src/backend/postmaster/postmaster.c | 10 +
src/backend/utils/activity/backend_status.c | 30 +
src/backend/utils/adt/pgstatfuncs.c | 50 +-
src/backend/utils/misc/guc.c | 30 +
src/bin/pgbench/pgbench.c | 17 +-
src/bin/psql/command.c | 23 +
src/common/Makefile | 4 +-
src/common/z_stream.c | 659 +++++++++++
src/common/zpq_stream.c | 1001 +++++++++++++++++
src/include/catalog/pg_proc.dat | 18 +-
src/include/common/z_stream.h | 107 ++
src/include/common/zpq_stream.h | 117 ++
src/include/libpq/libpq-be.h | 3 +
src/include/libpq/libpq.h | 1 +
src/include/libpq/pqcomm.h | 3 +
src/include/utils/backend_status.h | 7 +
src/interfaces/libpq/Makefile | 14 +
src/interfaces/libpq/exports.txt | 3 +
src/interfaces/libpq/fe-connect.c | 92 +-
src/interfaces/libpq/fe-exec.c | 10 +-
src/interfaces/libpq/fe-misc.c | 90 +-
src/interfaces/libpq/fe-protocol3.c | 56 +
src/interfaces/libpq/libpq-fe.h | 5 +
src/interfaces/libpq/libpq-int.h | 14 +
src/test/regress/expected/rules.out | 14 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
34 files changed, 2820 insertions(+), 89 deletions(-)
create mode 100644 src/common/z_stream.c
create mode 100644 src/common/zpq_stream.c
create mode 100644 src/include/common/z_stream.h
create mode 100644 src/include/common/zpq_stream.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7720ab9c58..5e4f38871b 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9509,7 +9509,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
+HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, compression, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index afbb6c35e3..8840edf913 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1035,6 +1035,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-libpq-compression" xreflabel="libpq_compression">
+ <term><varname>libpq_compression</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>libpq_compression</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter controls the available client-server traffic compression methods.
+ It allows rejecting compression requests even if it is supported by the server (for example, due to security, or CPU consumption).
+ The default is <literal>on</literal>, which means that all supported compression methods are allowed.
+ For more precise control, a list of the allowed compression methods can be specified.
+ For example, to allow only <literal>lz4</literal> and <literal>zlib</literal>, set the setting to <literal>lz4,zlib</literal>.
+ Also, maximal allowed compression level can be specified for each method, e.g. <literal>lz4:1,zlib:2</literal> setting will set the
+ maximal compression level for <literal>lz4</literal> to 1 and <literal>zlib</literal> to 2.
+ If a client requests the compression with a higher compression level, it will be set to the maximal allowed one.
+ Default (and recommended) maximal compression level for each algorithm is 1.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 14f35d37f6..5a16b4b64b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1260,6 +1260,43 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. The client sends a request with
+ a list of compression algorithms. Compression can be requested by a client
+ by including the <literal>"compression"</literal> option in its connection string.
+ This can either be a boolean value to enable or disable compression
+ (<literal>"true"</literal>/<literal>"false"</literal>,
+ <literal>"on"</literal>/<literal>"off"</literal>,
+ <literal>"yes"</literal>/<literal>"no"</literal>,
+ <literal>"1"</literal>/<literal>"0"</literal>),
+ <literal>"any"</literal>,
+ or an explicit list of comma-separated compression algorithms
+ which can optionally include compression level (<literal>"zlib,lz4:2"</literal>).
+ If compression is enabled but an algorithm is not explicitly specified,
+ the client library sends its full list of supported algorithms.
+ The server intersects the received compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ </para>
+ <para>
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ Compressed data is transmitted via the CompressedData messages.
+ </para>
+ <para>
+ Support for compression algorithms must be enabled when the server is compiled.
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option). In both cases, streaming mode is used.
+ By default, compression is not requested by the client.
+ Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..332041030a 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ It is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especially useful for importing/exporting data to/from the database using the <literal>COPY</literal> command
+ and for replication (both physical and logical). Compression can also reduce the server's response time
+ for queries returning a large amount of data (for example, JSON, BLOBs, text, ...).
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option).
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -262,6 +271,21 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CompressionAck</term>
+ <listitem>
+ <para>
+ The server accepts the client's compression request.
+ Compression is requested when a client connection includes the "compression" option, which contains a list of the requested compression algorithms.
+ The server intersects the requested compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>AuthenticationOk</term>
<listitem>
@@ -3577,6 +3601,141 @@ following:
</varlistentry>
+<varlistentry>
+<term>
+CompressionAck (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ List of the negotiated compression algorithms.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
+<varlistentry>
+<term>
+SetCompressionMethod (F & B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('k')
+</term>
+<listitem>
+<para>
+ Switch the current compression algorithm. Following CompressedData messages will be compressed via the compressed algorithm specified in this message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32(5)
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ Index of the new compression algorithm.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+CompressedData (F & B)
+</term>
+<listitem>
+<para>
+
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('m')
+</term>
+<listitem>
+<para>
+ Identifies the message as compressed data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ Compressed message data.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
AuthenticationSASLContinue (B)
@@ -6142,6 +6301,21 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>_pq_.compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. The value is a list of compression algorithms requested by the client with an optional
+ specification of compression level: <literal>"zlib,lz4:5"</literal>.
+ If the server does not support compression, the backend will ignore the _pq_.compression
+ parameter and will not send the CompressionAck message to the frontend.
+ By default, compression is disabled. Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 05c54b27de..79ed589ce9 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_lz4 = @with_lz4@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 0da848b1fd..f65f16fc48 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 61b515cdb8..8e91dc0cea 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -844,6 +844,15 @@ CREATE VIEW pg_stat_activity AS
LEFT JOIN pg_database AS D ON (S.datid = D.oid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
+CREATE VIEW pg_stat_network_traffic AS
+ SELECT
+ S.pid,
+ S.rx_raw_bytes,
+ S.tx_raw_bytes,
+ S.rx_compressed_bytes,
+ S.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL) AS S;
+
CREATE VIEW pg_stat_replication AS
SELECT
S.pid,
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 4c37df09cf..bac4d9f14b 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -60,6 +60,7 @@
#include <signal.h>
#include <fcntl.h>
#include <grp.h>
+#include <pgstat.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/socket.h>
@@ -77,11 +78,14 @@
#include "common/ip.h"
#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
#include "miscadmin.h"
#include "port/pg_bswap.h"
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/builtins.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -108,6 +112,9 @@
int Unix_socket_permissions;
char *Unix_socket_group;
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms;
+
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -130,6 +137,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream * PqStream;
+
+
/*
* Message status
*/
@@ -167,6 +177,145 @@ const PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+static ssize_t
+write_compressed(void *arg, void const *data, size_t size)
+{
+ ssize_t rc = secure_write((Port *) arg, (void *) data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, 0, rc);
+ return rc;
+}
+
+static ssize_t
+read_compressed(void *arg, void *data, size_t size)
+{
+ ssize_t rc = secure_read((Port *) arg, data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, rc, 0);
+ return rc;
+}
+
+/*
+ * Send the chosen compression algorithms to the client
+ */
+static void
+SendCompressionACK(char *compressors_str)
+{
+ StringInfoData buf;
+
+ pq_beginmessage(&buf, 'z');
+ pq_sendstring(&buf, compressors_str);
+ pq_endmessage(&buf);
+ pq_flush();
+}
+
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port *port)
+{
+ zpq_compressor *server_compressors;
+ zpq_compressor *client_compressors;
+ zpq_compressor *res_compressors;
+ size_t n_client_compressors;
+ size_t n_server_compressors;
+ size_t n_res_compressors = 0;
+ char *res_compressors_str;
+ char *client_compression_algorithms = port->compression_algorithms;
+
+ if (!client_compression_algorithms || !libpq_compress_algorithms)
+ {
+ return 0;
+ }
+
+ if (!zpq_parse_compression_setting(libpq_compress_algorithms, &server_compressors, &n_server_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms)));
+ return 0;
+ }
+
+ if (n_server_compressors == 0)
+ {
+ /*
+ * No enabled server compressors available, abort the compression
+ * initialization.
+ */
+ /* Send the compression acknowledgment with empty compressors list. */
+ char *empty_response = "";
+
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ if (!zpq_deserialize_compressors(client_compression_algorithms, &client_compressors, &n_client_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse received client compression methods: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ if (n_client_compressors == 0)
+ {
+ /*
+ * client did not provide any compression algorithms that server is
+ * compiled to support
+ */
+ ereport(LOG, (errmsg("server doesn't support any of the compression methods requested by the client: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ res_compressors = malloc(Max(n_client_compressors, n_server_compressors) * sizeof(zpq_compressor));
+
+ /*
+ * Intersect client and server compressors to determine the final list of
+ * the supported compressors. O(N^2) is negligible because of a small
+ * number of the compression methods.
+ */
+ for (size_t i = 0; i < n_client_compressors; i++)
+ {
+ for (size_t j = 0; j < n_server_compressors; j++)
+ {
+ if (client_compressors[i].impl == server_compressors[j].impl)
+ {
+ res_compressors[n_res_compressors].impl = client_compressors[i].impl;
+ /* prefer the lower compression level */
+ res_compressors[n_res_compressors++].level = Min(client_compressors[i].level, server_compressors[j].level);
+ break;
+ }
+ }
+ }
+ free(client_compressors);
+ free(server_compressors);
+
+ if (n_res_compressors == 0)
+ {
+ char *empty_response = "";
+
+ ereport(LOG, (errmsg("did not find any matches between the client and server compression methods, not enabling compression")));
+ free(res_compressors);
+ /* send the compression ack with empty compressors list */
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ res_compressors_str = zpq_serialize_compressors(res_compressors, n_res_compressors);
+ SendCompressionACK(res_compressors_str);
+
+ /* Init compression */
+ PqStream = zpq_create(res_compressors, n_res_compressors, write_compressed, read_compressed, MyProcPort, NULL, 0);
+ if (!PqStream)
+ {
+ ereport(LOG, (errmsg("failed to initialize the compression stream")));
+ return -1;
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -270,6 +419,9 @@ socket_close(int code, Datum arg)
}
#endif /* ENABLE_GSS */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -941,12 +1093,15 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * nowait parameter toggles non-blocking mode.
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -962,21 +1117,40 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ /*
+ * If streaming compression is enabled then use correspondent
+ * compression read function.
+ */
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
if (r < 0)
{
+ if (r == ZS_DECOMPRESS_ERROR)
+ {
+ char const *msg = zpq_decompress_error(PqStream);
+
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -997,7 +1171,8 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ pgstat_report_network_traffic(r, 0, 0, 0);
+ return r;
}
}
@@ -1012,7 +1187,8 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1031,7 +1207,8 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1048,48 +1225,15 @@ pq_peekbyte(void)
int
pq_getbyte_if_available(unsigned char *c)
{
- int r;
+ int r = 0;
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1110,7 +1254,8 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1144,7 +1289,8 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1168,7 +1314,6 @@ pq_buffer_has_data(void)
return (PqRecvPointer < PqRecvLength);
}
-
/* --------------------------------
* pq_startmsgread - begin reading a message from the client.
*
@@ -1372,13 +1517,24 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered_tx(PqStream) != 0)
+
+ /*
+ * has more data to flush or unsent data in internal compression
+ * buffer
+ */
{
int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
- if (r <= 0)
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1421,12 +1577,12 @@ internal_flush(void)
InterruptPending = 1;
return EOF;
}
+ pgstat_report_network_traffic(0, r, 0, 0);
last_reported_send_errno = 0; /* reset after any successful send */
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
@@ -1443,7 +1599,7 @@ socket_flush_if_writable(void)
int res;
/* Quick exit if nothing to do */
- if (PqSendPointer == PqSendStart)
+ if ((PqSendPointer == PqSendStart) && (zpq_buffered_tx(PqStream) == 0))
return 0;
/* No-op if reentrant call */
@@ -1466,7 +1622,7 @@ socket_flush_if_writable(void)
static bool
socket_is_send_pending(void)
{
- return (PqSendStart < PqSendPointer);
+ return (PqSendStart < PqSendPointer || (zpq_buffered_tx(PqStream) != 0));
}
/* --------------------------------
@@ -1989,3 +2145,16 @@ pq_check_connection(void)
return true;
}
+
+PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+
+Datum
+pg_compression_algorithm(PG_FUNCTION_ARGS)
+{
+ char const *algorithm_name = PqStream ? zpq_compress_algorithm_name(PqStream) : NULL;
+
+ if (algorithm_name)
+ PG_RETURN_TEXT_P(cstring_to_text(algorithm_name));
+ else
+ PG_RETURN_NULL();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 328ecafa8c..3576869311 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2233,6 +2233,8 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "_pq_.compression") == 0)
+ port->compression_algorithms = pstrdup(valptr);
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4555,6 +4557,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 7229598822..3682890304 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -338,6 +338,9 @@ pgstat_bestart(void)
lbeentry.st_xact_start_timestamp = 0;
lbeentry.st_databaseid = MyDatabaseId;
+ lbeentry.st_tx_raw_bytes = lbeentry.st_rx_raw_bytes =
+ lbeentry.st_tx_compressed_bytes = lbeentry.st_rx_compressed_bytes = 0;
+
/* We have userid for client-backends, wal-sender and bgworker processes */
if (lbeentry.st_backendType == B_BACKEND
|| lbeentry.st_backendType == B_WAL_SENDER
@@ -716,6 +719,33 @@ pgstat_report_xact_timestamp(TimestampTz tstamp)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Report current transaction start timestamp as the specified value.
+ * Zero means there is no active transaction.
+ */
+void
+pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ if (!pgstat_track_activities || !beentry)
+ return;
+
+ /*
+ * Update my status entry, following the protocol of bumping
+ * st_changecount before and after. We use a volatile pointer here to
+ * ensure the compiler doesn't try to get cute.
+ */
+ PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
+
+ beentry->st_rx_raw_bytes += rx_raw_bytes;
+ beentry->st_tx_raw_bytes += tx_raw_bytes;
+ beentry->st_rx_compressed_bytes += rx_compressed_bytes;
+ beentry->st_tx_compressed_bytes += tx_compressed_bytes;
+
+ PGSTAT_END_WRITE_ACTIVITY(beentry);
+}
+
/* ----------
* pgstat_read_current_status() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index f529c1561a..570d1b44c5 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -567,7 +567,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
Datum
pg_stat_get_activity(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_ACTIVITY_COLS 30
+#define PG_STAT_GET_ACTIVITY_COLS 34
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
int pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
@@ -916,6 +916,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[29] = true;
else
values[29] = UInt64GetDatum(beentry->st_query_id);
+ values[30] = beentry->st_rx_raw_bytes;
+ values[31] = beentry->st_tx_raw_bytes;
+ values[32] = beentry->st_rx_compressed_bytes;
+ values[33] = beentry->st_tx_compressed_bytes;
}
else
{
@@ -944,6 +948,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[27] = true;
nulls[28] = true;
nulls[29] = true;
+ nulls[30] = true;
+ nulls[31] = true;
+ nulls[32] = true;
+ nulls[33] = true;
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -1143,6 +1151,46 @@ pg_stat_get_backend_start(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(result);
}
+Datum
+pg_stat_get_network_traffic(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_NETWORK_TRAFFIC_COLS 4
+ TupleDesc tupdesc;
+ Datum values[PG_STAT_NETWORK_TRAFFIC_COLS];
+ bool nulls[PG_STAT_NETWORK_TRAFFIC_COLS];
+ int32 beid = PG_GETARG_INT32(0);
+ PgBackendStatus *beentry;
+
+ if ((beentry = pgstat_fetch_stat_beentry(beid)) == NULL)
+ PG_RETURN_NULL();
+ else if (!HAS_PGSTAT_PERMISSIONS(beentry->st_userid))
+ PG_RETURN_NULL();
+
+ /* Initialise values and NULL flags arrays */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(PG_STAT_NETWORK_TRAFFIC_COLS);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "rx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "tx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "rx_compressed_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "tx_compressed_bytes",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ /* Fill values and NULLs */
+ values[0] = beentry->st_rx_raw_bytes;
+ values[1] = beentry->st_tx_raw_bytes;
+ values[2] = beentry->st_rx_compressed_bytes;
+ values[3] = beentry->st_tx_compressed_bytes;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
Datum
pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index f9504d3aec..29214fdf45 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -51,6 +51,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "common/string.h"
+#include "common/zpq_stream.h"
#include "funcapi.h"
#include "jit/jit.h"
#include "libpq/auth.h"
@@ -191,6 +192,7 @@ static bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
static bool check_log_stats(bool *newval, void **extra, GucSource source);
static bool check_canonical_path(char **newval, void **extra, GucSource source);
static bool check_timezone_abbreviations(char **newval, void **extra, GucSource source);
+static bool check_libpq_compression(char **newval, void **extra, GucSource source);
static void assign_timezone_abbreviations(const char *newval, void *extra);
static void pg_timezone_abbrev_initialize(void);
static const char *show_archive_command(void);
@@ -4581,6 +4583,16 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"libpq_compression", PGC_SIGHUP, CLIENT_CONN_OTHER,
+ gettext_noop("Sets the list of allowed libpq compression algorithms."),
+ NULL
+ },
+ &libpq_compress_algorithms,
+ "on",
+ check_libpq_compression, NULL, NULL
+ },
+
{
{"application_name", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Sets the application name to be reported in statistics and logs."),
@@ -5013,6 +5025,8 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
@@ -11845,6 +11859,22 @@ check_ssl(bool *newval, void **extra, GucSource source)
return true;
}
+static bool
+check_libpq_compression(char **newval, void **extra, GucSource source)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(*newval, &compressors, &n_compressors))
+ {
+ GUC_check_errdetail("Cannot parse the libpq_compression setting.");
+ return false;
+ }
+
+ free(compressors);
+ return true;
+}
+
static bool
check_stage_log_stats(bool *newval, void **extra, GucSource source)
{
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ea9639984c..6d5dcb2b79 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6661,6 +6661,9 @@ threadRun(void *arg)
int nsocks; /* number of sockets to be waited for */
pg_time_usec_t min_usec;
pg_time_usec_t now = 0; /* set this only if needed */
+ bool buffered_rx = false; /* true if some of the clients has
+ * data left in SSL/ZPQ read
+ * buffers */
/*
* identify which client sockets should be checked for input, and
@@ -6695,6 +6698,9 @@ threadRun(void *arg)
*/
int sock = PQsocket(st->con);
+ /* check if conn has buffered SSL / ZPQ read data */
+ buffered_rx = buffered_rx || PQreadPending(st->con);
+
if (sock < 0)
{
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
@@ -6738,7 +6744,7 @@ threadRun(void *arg)
{
if (nsocks > 0)
{
- rc = wait_on_socket_set(sockets, min_usec);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, min_usec);
}
else /* nothing active, simple sleep */
{
@@ -6747,7 +6753,7 @@ threadRun(void *arg)
}
else /* no explicit delay, wait without timeout */
{
- rc = wait_on_socket_set(sockets, 0);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, 0);
}
if (rc < 0)
@@ -6786,8 +6792,11 @@ threadRun(void *arg)
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
goto done;
}
-
- if (!socket_has_input(sockets, sock, nsocks++))
+ if (PQreadPending(st->con))
+ {
+ nsocks++;
+ }
+ else if (!socket_has_input(sockets, sock, nsocks++))
continue;
}
else if (st->state == CSTATE_FINISHED ||
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index fb3bab9494..c87d86db58 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -171,6 +171,7 @@ static void print_with_linenumbers(FILE *output, char *lines,
const char *header_keyword);
static void minimal_error_message(PGresult *res);
+static void printCompressionInfo(void);
static void printSSLInfo(void);
static void printGSSInfo(void);
static bool printPsetInfo(const char *param, printQueryOpt *popt);
@@ -633,6 +634,7 @@ exec_command_conninfo(PsqlScanState scan_state, bool active_branch)
printf(_("You are connected to database \"%s\" as user \"%s\" on host \"%s\" at port \"%s\".\n"),
db, PQuser(pset.db), host, PQport(pset.db));
}
+ printCompressionInfo();
printSSLInfo();
printGSSInfo();
}
@@ -3647,6 +3649,27 @@ connection_warnings(bool in_startup)
}
}
+/*
+ * printCompressionInfo
+ *
+ * Print information about used compressor/decompressor
+ */
+static void
+printCompressionInfo(void)
+{
+ char *compressor = PQcompressor(pset.db);
+ char *decompressor = PQdecompressor(pset.db);
+
+ if (compressor != NULL)
+ {
+ printf(_("Compressor %s\n"), compressor);
+ }
+
+ if (decompressor != NULL)
+ {
+ printf(_("Decompressor %s\n"), decompressor);
+ }
+}
/*
* printSSLInfo
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..185c96063f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,7 +78,9 @@ OBJS_COMMON = \
unicode_norm.o \
username.o \
wait_error.o \
- wchar.o
+ wchar.o \
+ z_stream.o \
+ zpq_stream.o
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
new file mode 100644
index 0000000000..d94914e7dd
--- /dev/null
+++ b/src/common/z_stream.c
@@ -0,0 +1,659 @@
+#include "c.h"
+#include "pg_config.h"
+#include "common/z_stream.h"
+
+/*
+ * Functions implementing streaming compression algorithm
+ */
+typedef struct
+{
+ /*
+ * Name of compression algorithm.
+ */
+ char const *(*name) (void);
+
+ /*
+ * Create new compression stream. level: compression level
+ */
+ void *(*create_compressor) (int level);
+
+ /*
+ * Create new decompression stream.
+ */
+ void *(*create_decompressor) ();
+
+ /*
+ * Decompress up to "src_size" compressed bytes from *src and write up to
+ * "dst_size" raw (decompressed) bytes to *dst. Number of decompressed
+ * bytes written to *dst is stored in *dst_processed. Number of compressed
+ * bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during decompression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * decompressor internal buffers.
+ *
+ * ZS_STREAM_END if encountered end of compressed data stream.
+ *
+ * ZS_DECOMPRESS_ERROR if encountered an error during decompression
+ * attempt.
+ */
+ ssize_t (*decompress) (void *ds, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Compress up to "src_size" raw (non-compressed) bytes from *src and
+ * write up to "dst_size" compressed bytes to *dst. Number of compressed
+ * bytes written to *dst is stored in *dst_processed. Number of
+ * non-compressed bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during compression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * compressor internal buffers.
+ *
+ * ZS_COMPRESS_ERROR if encountered an error during compression attempt.
+ */
+ ssize_t (*compress) (void *cs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Free compression stream created by create_compressor function.
+ */
+ void (*free_compressor) (void *cs);
+
+ /*
+ * Free decompression stream created by create_decompressor function.
+ */
+ void (*free_decompressor) (void *ds);
+
+ /*
+ * Get compressor error message.
+ */
+ char const *(*compress_error) (void *cs);
+
+ /*
+ * Get decompressor error message.
+ */
+ char const *(*decompress_error) (void *ds);
+
+ ssize_t (*end_compression) (void *cs, void *dst, size_t dst_size, size_t *dst_processed);
+} ZAlgorithm;
+
+struct ZStream
+{
+ ZAlgorithm const *algorithm;
+ void *stream;
+ bool not_flushed;
+};
+
+#if HAVE_LIBZ
+
+#include <stdlib.h>
+#include <zlib.h>
+
+
+static void *
+zlib_create_compressor(int level)
+{
+ int rc;
+ z_stream *c_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(c_stream, 0, sizeof(*c_stream));
+ rc = deflateInit(c_stream, level);
+ if (rc != Z_OK)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+zlib_create_decompressor()
+{
+ int rc;
+ z_stream *d_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(d_stream, 0, sizeof(*d_stream));
+ rc = inflateInit(d_stream);
+ if (rc != Z_OK)
+ {
+ free(d_stream);
+ return NULL;
+ }
+ return d_stream;
+}
+
+static ssize_t
+zlib_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *ds = (z_stream *) d_stream;
+ int rc;
+
+ ds->next_in = (Bytef *) src;
+ ds->avail_in = src_size;
+ ds->next_out = (Bytef *) dst;
+ ds->avail_out = dst_size;
+
+ rc = inflate(ds, Z_SYNC_FLUSH);
+ *src_processed = src_size - ds->avail_in;
+ *dst_processed = dst_size - ds->avail_out;
+
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_STREAM_END;
+ }
+ if (rc != Z_OK && rc != Z_BUF_ERROR)
+ {
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+ unsigned deflate_pending = 0;
+
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = (Bytef *) src;
+ cs->avail_in = src_size;
+
+ rc = deflate(cs, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ *dst_processed = dst_size - cs->avail_out;
+ *src_processed = src_size - cs->avail_in;
+
+ deflatePending(cs, &deflate_pending, Z_NULL); /* check if any data left
+ * in deflate buffer */
+ if (deflate_pending > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+zlib_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = NULL;
+ cs->avail_in = 0;
+
+ rc = deflate(cs, Z_STREAM_END);
+ Assert(rc == Z_OK || rc == Z_STREAM_END);
+ *dst_processed = dst_size - cs->avail_out;
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_OK;
+ }
+
+ return ZS_DATA_PENDING;
+}
+
+static void
+zlib_free_compressor(void *c_stream)
+{
+ z_stream *cs = (z_stream *) c_stream;
+
+ if (cs != NULL)
+ {
+ deflateEnd(cs);
+ free(cs);
+ }
+}
+
+static void
+zlib_free_decompressor(void *d_stream)
+{
+ z_stream *ds = (z_stream *) d_stream;
+
+ if (ds != NULL)
+ {
+ inflateEnd(ds);
+ free(ds);
+ }
+}
+
+static char const *
+zlib_error(void *stream)
+{
+ z_stream *zs = (z_stream *) stream;
+
+ return zs->msg;
+}
+
+static char const *
+zlib_name(void)
+{
+ return "zlib";
+}
+
+#endif
+
+#if USE_LZ4
+#include <lz4.h>
+
+#define MESSAGE_MAX_BYTES 819200
+#define RING_BUFFER_BYTES (1024 * 64 + MESSAGE_MAX_BYTES)
+
+typedef struct ZS_LZ4_CStream
+{
+ LZ4_stream_t *stream;
+ int level;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_CStream;
+
+typedef struct ZS_LZ4_DStream
+{
+ LZ4_streamDecode_t *stream;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_DStream;
+
+static void *
+lz4_create_compressor(int level)
+{
+ ZS_LZ4_CStream *c_stream = (ZS_LZ4_CStream *) malloc(sizeof(ZS_LZ4_CStream));
+ if (c_stream == NULL) {
+ return NULL;
+ }
+ c_stream->stream = LZ4_createStream();
+ c_stream->level = level;
+ c_stream->buf_pos = 0;
+ if (c_stream->stream == NULL)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+lz4_create_decompressor()
+{
+ ZS_LZ4_DStream *d_stream = (ZS_LZ4_DStream *) malloc(sizeof(ZS_LZ4_DStream));
+ if (d_stream == NULL) {
+ return NULL;
+ }
+
+ d_stream->stream = LZ4_createStreamDecode();
+ d_stream->buf_pos = 0;
+ if (d_stream->stream == NULL)
+ {
+ free(d_stream);
+ return NULL;
+ }
+
+ return d_stream;
+}
+
+static ssize_t
+lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+ Assert(src_size > 0);
+
+ char* const decPtr = &ds->buf[ds->buf_pos];
+
+ const int decBytes = LZ4_decompress_safe_continue(
+ ds->stream, src, decPtr, (int) src_size, (int) dst_size);
+
+ Assert(decBytes > 0);
+
+ *dst_processed = decBytes;
+ *src_processed = src_size;
+
+ memcpy(dst, decPtr, decBytes); /* write msg length */
+
+ ds->buf_pos += decBytes;
+ if (ds->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ ds->buf_pos = 0;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ src_size = Min(MESSAGE_MAX_BYTES, src_size);
+
+ Assert(src_size > 0);
+
+ memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
+
+ Assert(dst_size >= LZ4_compressBound(src_size));
+
+ const int cmpBytes = LZ4_compress_fast_continue(
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, 1);
+
+ Assert(cmpBytes > 0);
+ Assert(cmpBytes <= MESSAGE_MAX_BYTES);
+
+ *dst_processed = cmpBytes;
+ *src_processed = src_size;
+
+ cs->buf_pos += src_size;
+ if (cs->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ cs->buf_pos = 0;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+lz4_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ *dst_processed = 0;
+ return ZS_OK;
+}
+
+static void
+lz4_free_compressor(void *c_stream)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ if (cs->stream != NULL) {
+ LZ4_freeStream(cs->stream);
+ }
+ free(cs);
+ }
+}
+
+static void
+lz4_free_decompressor(void *d_stream)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ if (ds->stream != NULL) {
+ LZ4_freeStreamDecode(ds->stream);
+ }
+ free(ds);
+ }
+}
+
+static char const *
+lz4_error(void *stream)
+{
+ // lz4 doesn't have any explicit API to get the error names
+ return "NO_MSG";
+}
+
+static char const *
+lz4_name(void)
+{
+ return "lz4";
+}
+
+#endif
+
+static char const *
+no_compression_name(void)
+{
+ return NULL;
+}
+
+/*
+ * Array with all supported compression algorithms.
+ */
+static ZAlgorithm const zs_algorithms[] =
+{
+#if HAVE_LIBZ
+ {zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
+#endif
+#if USE_LZ4
+ {lz4_name, lz4_create_compressor, lz4_create_decompressor, lz4_decompress, lz4_compress, lz4_free_compressor, lz4_free_decompressor, lz4_error, lz4_error, lz4_end},
+#endif
+ {no_compression_name}
+};
+
+inline bool
+zs_is_valid_impl_id(unsigned int id)
+{
+ return id >= 0 && id < (sizeof(zs_algorithms) / sizeof(*zs_algorithms));
+}
+
+static ssize_t
+zs_init_compressor(ZStream * zs, unsigned int c_alg_impl, int c_level)
+{
+ if (!zs_is_valid_impl_id(c_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[c_alg_impl];
+ zs->stream = zs->algorithm->create_compressor(c_level);
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+static ssize_t
+zs_init_decompressor(ZStream * zs, unsigned int d_alg_impl)
+{
+ if (!zs_is_valid_impl_id(d_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[d_alg_impl];
+ zs->stream = zs->algorithm->create_decompressor();
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Index of used compression algorithm in zs_algorithms array.
+ */
+ZStream *
+zs_create_compressor(unsigned int c_alg_impl, int c_level)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_compressor(zs, c_alg_impl, c_level))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ZStream *
+zs_create_decompressor(unsigned int d_alg_impl)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_decompressor(zs, d_alg_impl))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ssize_t
+zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *src_processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->decompress(zs->stream,
+ src, src_size, src_processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+
+ if (rc == ZS_OK || rc == ZS_INCOMPLETE_SRC || rc == ZS_STREAM_END) {
+ return rc;
+ }
+
+ return ZS_DECOMPRESS_ERROR;
+}
+
+ssize_t
+zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->compress(zs->stream,
+ buf, size, processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+void
+zs_compressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_compressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+void
+zs_decompressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_decompressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+ssize_t
+zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *dst_processed = 0;
+
+ rc = zs->algorithm->end_compression(zs->stream, dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+char const *
+zs_compress_error(ZStream * zs)
+{
+ return zs->algorithm->compress_error(zs->stream);
+}
+
+char const *
+zs_decompress_error(ZStream * zs)
+{
+ return zs->algorithm->decompress_error(zs->stream);
+}
+
+bool
+zs_buffered(ZStream * zs)
+{
+ return zs ? zs->not_flushed : 0;
+}
+
+
+/*
+ * Get list of the supported algorithms.
+ */
+char **
+zs_get_supported_algorithms(void)
+{
+ size_t n_algorithms = sizeof(zs_algorithms) / sizeof(*zs_algorithms);
+ char **algorithm_names = malloc(n_algorithms * sizeof(char *));
+
+ for (size_t i = 0; i < n_algorithms; i++)
+ {
+ algorithm_names[i] = (char *) zs_algorithms[i].name();
+ }
+
+ return algorithm_names;
+}
+
+char const *
+zs_compress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
+
+char const *
+zs_decompress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000000..294ad249ce
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,1001 @@
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+#include <unistd.h>
+#include <math.h>
+
+#include "common/zpq_stream.h"
+#include "pg_config.h"
+#include "port/pg_bswap.h"
+
+/* log warnings on backend */
+#ifndef FRONTEND
+#define pg_log_warning(...) elog(WARNING, __VA_ARGS__)
+#else
+#define pg_log_warning(...) (void)0
+#endif
+
+/* ZpqBuffer size, in bytes */
+#define ZPQ_BUFFER_SIZE 8192000
+/* CompressedData msg_type */
+#define ZPQ_COMPRESSED_MSG_TYPE 'm'
+/* SetCompressionMethod msg_type */
+#define ZPQ_SET_COMPRESSION_MSG_TYPE 'k'
+
+#define ZPQ_COMPRESS_THRESHOLD 60
+
+typedef struct ZpqBuffer ZpqBuffer;
+
+
+/* ZpqBuffer used as RX/TX buffer in ZpqStream */
+struct ZpqBuffer
+{
+ char buf[ZPQ_BUFFER_SIZE];
+ size_t size; /* current size of buf */
+ size_t pos; /* current position in buf, in range [0, size] */
+};
+
+static inline void
+zpq_buf_init(ZpqBuffer * zb)
+{
+ zb->size = 0;
+ zb->pos = 0;
+}
+
+static inline size_t
+zpq_buf_left(ZpqBuffer * zb)
+{
+ Assert(zb->buf);
+ return ZPQ_BUFFER_SIZE - zb->size;
+}
+
+static inline size_t
+zpq_buf_unread(ZpqBuffer * zb)
+{
+ return zb->size - zb->pos;
+}
+
+static inline char *
+zpq_buf_size(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->size;
+}
+
+static inline char *
+zpq_buf_pos(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->pos;
+}
+
+static inline void
+zpq_buf_size_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->size += value;
+}
+
+static inline void
+zpq_buf_pos_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->pos += value;
+}
+
+static inline void
+zpq_buf_reuse(ZpqBuffer * zb)
+{
+ size_t unread = zpq_buf_unread(zb);
+
+ if (unread > 5) /* can read message header, don't do anything */
+ return;
+ if (unread == 0)
+ {
+ zb->size = 0;
+ zb->pos = 0;
+ return;
+ }
+ memmove(zb->buf, zb->buf + zb->pos, unread);
+ zb->size = unread;
+ zb->pos = 0;
+}
+
+struct ZpqStream
+{
+ ZStream *c_stream; /* underlying compression stream */
+ ZStream *d_stream; /* underlying decompression stream */
+
+ size_t tx_total; /* amount of bytes sent to tx_func */
+
+ size_t tx_total_raw; /* amount of bytes received by zpq_write */
+ size_t rx_total; /* amount of bytes read by rx_func */
+ size_t rx_total_raw; /* amount of bytes returned by zpq_write */
+ bool is_compressing; /* current compression state */
+
+ bool is_decompressing; /* current decompression state */
+ size_t rx_msg_bytes_left; /* number of bytes left to process without
+ * changing the decompression state */
+ size_t tx_msg_bytes_left; /* number of bytes left to process without
+ * changing the compression state */
+
+ ZpqBuffer rx_in; /* buffer for unprocessed data read by rx_func */
+ ZpqBuffer tx_in; /* buffer for unprocessed data consumed by
+ * zpq_write */
+ ZpqBuffer tx_out; /* buffer for processed data waiting for send
+ * via tx_func */
+
+ zpq_rx_func rx_func;
+ zpq_tx_func tx_func;
+ void *arg;
+
+ zpq_compressor *compressors; /* compressors array holds the available
+ * compressors to use for
+ * compression/decompression */
+ size_t n_compressors; /* size of the compressors array */
+ int compress_alg_idx; /* index of the active compression
+ * algorithm */
+ int decompress_alg_idx; /* index of the active decompression
+ * algorithm */
+ int compressor_by_msg_type[256]; /* map to choose a compressor
+ * by the protocol message
+ * type */
+
+ bool reading_set_compression; /* utility marker indicating
+ * partial SetCompressionMethod
+ * read */
+};
+
+/*
+ * Message compression map defines the logic for choosing the compressor
+ * based on the protocol message type. Currently, it is a basic prototype to demonstrate
+ * the capabilities of the on-the-fly compression switch.
+ */
+static inline void
+zpq_build_msg_compression_map(ZpqStream * zpq)
+{
+ int i;
+
+ for (i = 0; i < 256; i++)
+ {
+ zpq->compressor_by_msg_type[i] = -1;
+ }
+
+ for (i = 0; i < zpq->n_compressors; i++)
+ {
+ /* compress CopyData, DataRow and Query messages */
+ if (zpq->compressor_by_msg_type['d'] == -1) {
+ zpq->compressor_by_msg_type['d'] = i;
+ }
+ if (zpq->compressor_by_msg_type['D'] == -1) {
+ zpq->compressor_by_msg_type['D'] = i;
+ }
+ if (zpq->compressor_by_msg_type['Q'] == -1) {
+ zpq->compressor_by_msg_type['Q'] = i;
+ }
+ }
+}
+
+/*
+ * Choose the index of compressor to use for the message of msg_type with msg_len.
+ * Return values:
+ * - the non-negative index of zpq->compressors array
+ * - -1, if message should not be compressed
+ */
+static inline int
+zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
+ {
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
+ }
+ return -1;
+}
+
+/*
+ * Check if should compress message of msg_type with msg_len.
+ * Return true if should, false if should not.
+ */
+static inline bool
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
+}
+
+/*
+ * Check if message is a CompressedData.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_compressed_msg(char msg_type)
+{
+ return msg_type == ZPQ_COMPRESSED_MSG_TYPE;
+}
+
+/*
+ * Check if message is a SetCompressionMethod.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_set_compression_msg(char msg_type)
+{
+ return msg_type == ZPQ_SET_COMPRESSION_MSG_TYPE;
+}
+
+ZpqStream *
+zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size)
+{
+ ZpqStream *zpq;
+
+ /* zpqStream needs at least one compressor */
+ if (n_compressors == 0 || compressors == NULL)
+ {
+ return NULL;
+ }
+ zpq = (ZpqStream *) malloc(sizeof(ZpqStream));
+
+ zpq->compressors = compressors;
+ zpq->n_compressors = n_compressors;
+ zpq->compress_alg_idx = -1;
+ zpq->decompress_alg_idx = -1;
+
+ zpq->is_compressing = false;
+ zpq->is_decompressing = false;
+ zpq->rx_msg_bytes_left = 0;
+ zpq->tx_msg_bytes_left = 0;
+ zpq_buf_init(&zpq->tx_in);
+
+ zpq->tx_total = 0;
+ zpq->tx_total_raw = 0;
+ zpq->rx_total = 0;
+ zpq->rx_total_raw = 0;
+
+ zpq_buf_init(&zpq->rx_in);
+ zpq_buf_size_advance(&zpq->rx_in, rx_data_size);
+ Assert(rx_data_size < ZPQ_BUFFER_SIZE);
+ memcpy(zpq->rx_in.buf, rx_data, rx_data_size);
+
+ zpq_buf_init(&zpq->tx_out);
+
+ zpq->rx_func = rx_func;
+ zpq->tx_func = tx_func;
+ zpq->arg = arg;
+ zpq->reading_set_compression = false;
+ zpq->c_stream = NULL;
+ zpq->d_stream = NULL;
+
+ zpq_build_msg_compression_map(zpq);
+
+ return zpq;
+}
+
+/* Compress up to src_size bytes from *src into CompressedData and write it to the tx buffer.
+ * Returns ZS_OK on success, ZS_COMPRESS_ERROR if encountered a compression error. */
+static inline ssize_t
+zpq_write_compressed_message(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ size_t compressed_len;
+ ssize_t rc;
+ uint32 size;
+
+ /* check if have enough space */
+ if (zpq_buf_left(&zpq->tx_out) <= 5)
+ {
+ /* too little space for CompressedData, abort */
+ *src_processed = 0;
+ return ZS_OK;
+ }
+
+ compressed_len = 0;
+ rc = zs_write(zpq->c_stream, src, src_size, src_processed,
+ zpq_buf_size(&zpq->tx_out) + 5, zpq_buf_left(&zpq->tx_out) - 5, &compressed_len);
+
+ if (compressed_len > 0)
+ {
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_COMPRESSED_MSG_TYPE;
+ size = pg_hton32(compressed_len + 4);
+
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &size, sizeof(uint32)); /* write msg length */
+ compressed_len += 5; /* append header length to compressed data
+ * length */
+ }
+
+ zpq->tx_total_raw += *src_processed;
+ zpq->tx_total += compressed_len;
+ zpq_buf_size_advance(&zpq->tx_out, compressed_len);
+ return rc;
+}
+
+/* Copy the data directly from *src to the tx buffer */
+static void
+zpq_write_uncompressed(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ src_size = Min(zpq_buf_left(&zpq->tx_out), src_size);
+ memcpy(zpq_buf_size(&zpq->tx_out), src, src_size);
+
+ zpq->tx_total_raw += src_size;
+ zpq->tx_total += src_size;
+ zpq_buf_size_advance(&zpq->tx_out, src_size);
+ *src_processed = src_size;
+}
+
+static ssize_t
+zpq_write_set_compression_msg(ZpqStream * zpq, int new_compress_idx)
+{
+ uint32 len;
+ uint8 idx;
+
+ /*
+ * check if have enough space: msg_type(1 byte) + msg_len(4 bytes) +
+ * compress_alg_idx(1 byte)
+ */
+ if (zpq_buf_left(&zpq->tx_out) < 6)
+ {
+ return -1;
+ }
+
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_SET_COMPRESSION_MSG_TYPE;
+ len = pg_hton32(5);
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &len, sizeof(uint32)); /* write msg length */
+
+ /* currently we expect idx to be in range [0, 255] */
+ Assert(new_compress_idx >= 0 && new_compress_idx <= UINT8_MAX);
+ idx = (uint8) new_compress_idx;
+ memcpy(zpq_buf_size(&zpq->tx_out) + 5, &idx, sizeof(uint8)); /* write
+ * new_compress_idx */
+
+ zpq->tx_total_raw += 6;
+ zpq->tx_total += 6;
+ zpq_buf_size_advance(&zpq->tx_out, 6);
+ return 0;
+}
+
+/* Determine if should compress the next message and change the current compression state */
+static ssize_t
+zpq_toggle_compression(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ int new_compress_idx = zpq_choose_compressor(zpq, msg_type, msg_len);
+ bool should_compress = new_compress_idx != -1;
+
+ /*
+ * negative new_compress_idx indicates that we should not compress this
+ * message
+ */
+ if (should_compress)
+ {
+ /*
+ * if the new compressor does not match the current one, process the
+ * switch
+ */
+ if (zpq->compress_alg_idx != new_compress_idx)
+ {
+ if (zpq_write_set_compression_msg(zpq, new_compress_idx))
+ {
+ /*
+ * come back later when we can write the entire
+ * SetCompressionMethod message
+ */
+ return 0;
+ }
+
+ zs_compressor_free(zpq->c_stream);
+ zpq->c_stream = zs_create_compressor(zpq->compressors[new_compress_idx].impl, zpq->compressors[new_compress_idx].level);
+ if (zpq->c_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->compress_alg_idx = new_compress_idx;
+ }
+ }
+
+ zpq->is_compressing = should_compress;
+ zpq->tx_msg_bytes_left = msg_len + 1;
+ return 0;
+}
+
+/*
+ * Internal write function. Reads the data from *src buffer,
+ * determines the postgres messages type and length.
+ * If message matches the compression criteria, it wraps the message into
+ * CompressedData. Otherwise, leaves the message unchanged.
+ * If *src data ends with incomplete message header, this function is not
+ * going to read this message header.
+ * Returns number of written raw bytes or error code.
+ * In the last case number of bytes written is stored in *processed.
+ */
+static ssize_t
+zpq_write_internal(ZpqStream * zpq, void const *src, size_t src_size, size_t *processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ do
+ {
+ /*
+ * try to read ahead the next message types and increase
+ * tx_msg_bytes_left, if possible
+ */
+ while (zpq->tx_msg_bytes_left > 0 && src_size - src_pos >= zpq->tx_msg_bytes_left + 5)
+ {
+ char msg_type = *((char *) src + src_pos + zpq->tx_msg_bytes_left);
+ uint32 msg_len;
+
+ memcpy(&msg_len, (char *) src + src_pos + zpq->tx_msg_bytes_left + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ if (zpq_should_compress(zpq, msg_type, msg_len) != zpq->is_compressing)
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+ zpq->tx_msg_bytes_left += msg_len + 1;
+ }
+
+ /*
+ * Write CompressedData if currently is compressing or have some
+ * buffered data left in underlying compression stream
+ */
+ if (zs_buffered(zpq->c_stream) || (zpq->is_compressing && zpq->tx_msg_bytes_left > 0))
+ {
+ size_t buf_processed = 0;
+ size_t to_compress = Min(zpq->tx_msg_bytes_left, src_size - src_pos);
+
+ rc = zpq_write_compressed_message(zpq, (char *) src + src_pos, to_compress, &buf_processed);
+ src_pos += buf_processed;
+ zpq->tx_msg_bytes_left -= buf_processed;
+
+ if (rc != ZS_OK)
+ {
+ *processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * If not going to compress the data from *src, just write it
+ * uncompressed.
+ */
+ else if (zpq->tx_msg_bytes_left > 0)
+ { /* determine next message type */
+ size_t copy_len = Min(src_size - src_pos, zpq->tx_msg_bytes_left);
+ size_t copy_processed = 0;
+
+ zpq_write_uncompressed(zpq, (char *) src + src_pos, copy_len, ©_processed);
+ src_pos += copy_processed;
+ zpq->tx_msg_bytes_left -= copy_processed;
+ }
+
+ /*
+ * Reached the compression toggle point, fetch next message header to
+ * determine compression state.
+ */
+ else
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ if (src_size - src_pos < 5)
+ {
+ /*
+ * must return here because we can't continue without full
+ * message header
+ */
+ *processed = src_pos;
+ return ZPQ_INCOMPLETE_HEADER;
+ }
+
+ msg_type = *((char *) src + src_pos);
+ memcpy(&msg_len, (char *) src + src_pos + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ rc = zpq_toggle_compression(zpq, msg_type, msg_len);
+ if (rc)
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * repeat sending while there is some data in input or internal
+ * compression buffer
+ */
+ } while (src_pos < src_size && zpq_buf_left(&zpq->tx_out) > 6);
+
+ return src_pos;
+}
+
+ssize_t
+zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ /* try to process as much data as possible before calling the tx_func */
+ while (zpq_buf_left(&zpq->tx_out) > 6)
+ {
+ size_t copy_len = Min(zpq_buf_left(&zpq->tx_in), src_size - src_pos);
+ size_t processed;
+
+ memcpy(zpq_buf_size(&zpq->tx_in), (char *) src + src_pos, copy_len);
+ zpq_buf_size_advance(&zpq->tx_in, copy_len);
+ src_pos += copy_len;
+
+ if (zpq_buf_unread(&zpq->tx_in) == 0 && !zs_buffered(zpq->c_stream))
+ {
+ break;
+ }
+
+ processed = 0;
+
+ rc = zpq_write_internal(zpq, zpq_buf_pos(&zpq->tx_in), zpq_buf_unread(&zpq->tx_in), &processed);
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, rc);
+ zpq_buf_reuse(&zpq->tx_in);
+ }
+ else
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, processed);
+ zpq_buf_reuse(&zpq->tx_in);
+ if (rc == ZPQ_INCOMPLETE_HEADER)
+ {
+ break;
+ }
+ *src_processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * call the tx_func if have any bytes to send
+ */
+ while (zpq_buf_unread(&zpq->tx_out))
+ {
+ rc = zpq->tx_func(zpq->arg, zpq_buf_pos(&zpq->tx_out), zpq_buf_unread(&zpq->tx_out));
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_out, rc);
+ }
+ else
+ {
+ *src_processed = src_pos;
+ zpq_buf_reuse(&zpq->tx_out);
+ return rc;
+ }
+ }
+
+ zpq_buf_reuse(&zpq->tx_out);
+ return src_pos;
+}
+
+/* Decompress bytes from RX buffer and write up to dst_len of uncompressed data to *dst.
+ * Returns:
+ * ZS_OK on success,
+ * ZS_STREAM_END if reached end of compressed chunk
+ * ZS_DECOMPRESS_ERROR if encountered a decompression error */
+static inline ssize_t
+zpq_read_compressed_message(ZpqStream * zpq, char *dst, size_t dst_len, size_t *dst_processed)
+{
+ size_t rx_processed = 0;
+ ssize_t rc;
+ size_t read_len = Min(zpq->rx_msg_bytes_left, zpq_buf_unread(&zpq->rx_in));
+
+ Assert(read_len == zpq->rx_msg_bytes_left);
+ rc = zs_read(zpq->d_stream, zpq_buf_pos(&zpq->rx_in), read_len, &rx_processed,
+ dst, dst_len, dst_processed);
+
+ zpq_buf_pos_advance(&zpq->rx_in, rx_processed);
+ zpq->rx_total_raw += *dst_processed;
+ zpq->rx_msg_bytes_left -= rx_processed;
+ return rc;
+}
+
+/* Copy up to dst_len bytes from rx buffer to *dst.
+ * Returns amount of bytes copied. */
+static inline size_t
+zpq_read_uncompressed(ZpqStream * zpq, char *dst, size_t dst_len)
+{
+ size_t copy_len;
+
+ Assert(zpq_buf_unread(&zpq->rx_in) > 0);
+ copy_len = Min(zpq->rx_msg_bytes_left, Min(zpq_buf_unread(&zpq->rx_in), dst_len));
+
+ memcpy(dst, zpq_buf_pos(&zpq->rx_in), copy_len);
+
+ zpq_buf_pos_advance(&zpq->rx_in, copy_len);
+ zpq->rx_total_raw += copy_len;
+ zpq->rx_msg_bytes_left -= copy_len;
+ return copy_len;
+}
+
+/* Determine if should decompress the next message and
+ * change the current decompression state */
+static inline void
+zpq_toggle_decompression(ZpqStream * zpq)
+{
+ uint32 msg_len;
+ char msg_type = *zpq_buf_pos(&zpq->rx_in);
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+
+ if (zpq_is_set_compression_msg(msg_type))
+ {
+ Assert(msg_len == 5);
+ zpq->reading_set_compression = true;
+ /* set compression message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ }
+ else
+ {
+ zpq->is_decompressing = zpq_is_compressed_msg(msg_type);
+ zpq->rx_msg_bytes_left = msg_len + 1;
+
+ if (zpq->is_decompressing)
+ {
+ /* compressed message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ zpq->rx_msg_bytes_left -= 5;
+ }
+ }
+}
+
+static inline ssize_t
+zpq_process_switch(ZpqStream * zpq)
+{
+ uint8 algorithm_idx;
+
+ if (zpq_buf_unread(&zpq->rx_in) < 1)
+ {
+ return 0;
+ }
+
+ algorithm_idx = *zpq_buf_pos(&zpq->rx_in);
+
+ zpq_buf_pos_advance(&zpq->rx_in, 1);
+ zpq->reading_set_compression = false;
+
+ if (algorithm_idx != zpq->decompress_alg_idx)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ zpq->d_stream = zs_create_decompressor(zpq->compressors[algorithm_idx].impl);
+ if (zpq->d_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->decompress_alg_idx = algorithm_idx;
+ }
+
+ return 0;
+}
+
+ssize_t
+zpq_read(ZpqStream * zpq, void *dst, size_t dst_size)
+{
+ size_t dst_pos = 0;
+ size_t dst_processed = 0;
+ ssize_t rc;
+
+ /* Read until some data fetched */
+ while (dst_pos == 0)
+ {
+ zpq_buf_reuse(&zpq->rx_in);
+
+ if (!zpq_buffered_rx(zpq) || (zpq->is_decompressing && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left))
+ {
+ rc = zpq->rx_func(zpq->arg, zpq_buf_size(&zpq->rx_in), zpq_buf_left(&zpq->rx_in));
+ if (rc > 0) /* read fetches some data */
+ {
+ zpq->rx_total += rc;
+ zpq_buf_size_advance(&zpq->rx_in, rc);
+ }
+ else /* read failed */
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * try to read ahead the next message types and increase
+ * rx_msg_bytes_left, if possible (ONLY UNCOMPRESSED MESSAGES)
+ */
+ while (!zpq->is_decompressing && zpq->rx_msg_bytes_left > 0 && (zpq_buf_unread(&zpq->rx_in) >= zpq->rx_msg_bytes_left + 5))
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ msg_type = *(zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left);
+ if (zpq_is_compressed_msg(msg_type) || zpq_is_set_compression_msg(msg_type))
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left + 1, 4);
+ zpq->rx_msg_bytes_left += pg_ntoh32(msg_len) + 1;
+ }
+
+
+ if (zpq->rx_msg_bytes_left > 0 || zs_buffered(zpq->d_stream))
+ {
+ dst_processed = 0;
+ if (zpq->is_decompressing || zs_buffered(zpq->d_stream))
+ {
+ if (!zs_buffered(zpq->d_stream) && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left) {
+ /* prefer to read only the fully compressed messages or read if some data is buffered */
+ continue;
+ }
+ rc = zpq_read_compressed_message(zpq, dst, dst_size - dst_pos, &dst_processed);
+ dst_pos += dst_processed;
+ if (rc == ZS_STREAM_END)
+ {
+ continue;
+ }
+ if (rc != ZS_OK)
+ {
+ return rc;
+ }
+ }
+ else
+ dst_pos += zpq_read_uncompressed(zpq, dst, dst_size - dst_pos);
+ }
+ else if (zpq->reading_set_compression)
+ {
+ zpq_process_switch(zpq);
+ }
+ else if (zpq_buf_unread(&zpq->rx_in) >= 5)
+ zpq_toggle_decompression(zpq);
+ }
+ return dst_pos;
+}
+
+bool
+zpq_buffered_rx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->rx_in) >= 5 || (zpq_buf_unread(&zpq->rx_in) > 0 && zpq->rx_msg_bytes_left > 0) ||
+ zs_buffered(zpq->d_stream) : 0;
+}
+
+bool
+zpq_buffered_tx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->tx_in) >= 5 || (zpq_buf_unread(&zpq->tx_in) > 0 && zpq->tx_msg_bytes_left > 0) || zpq_buf_unread(&zpq->tx_out) > 0 ||
+ zs_buffered(zpq->c_stream) : 0;
+}
+
+void
+zpq_free(ZpqStream * zpq)
+{
+ if (zpq)
+ {
+ if (zpq->c_stream)
+ {
+ zs_compressor_free(zpq->c_stream);
+ }
+ if (zpq->d_stream)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ }
+ free(zpq);
+ }
+}
+
+char const *
+zpq_compress_error(ZpqStream * zpq)
+{
+ return zs_compress_error(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_error(ZpqStream * zpq)
+{
+ return zs_decompress_error(zpq->d_stream);
+}
+
+char const *
+zpq_compress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_compress_algorithm_name(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_decompress_algorithm_name(zpq->d_stream);
+}
+
+bool
+zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int i;
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *protocol_extension = strchr(val, ';');
+
+ *compressors = NULL;
+ *n_compressors = 0;
+
+ /* No protocol extensions are currently supported */
+ if (protocol_extension)
+ *protocol_extension = '\0';
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ if (pg_strcasecmp(val, "true") == 0 ||
+ pg_strcasecmp(val, "yes") == 0 ||
+ pg_strcasecmp(val, "on") == 0 ||
+ pg_strcasecmp(val, "any") == 0 ||
+ pg_strcasecmp(val, "1") == 0)
+ {
+ /* return all available compressors */
+ *n_compressors = n_supported_algorithms;
+
+ if (n_supported_algorithms)
+ {
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+ for (i = 0; i < n_supported_algorithms; i++)
+ {
+ (*compressors)[i].impl = i;
+ (*compressors)[i].level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ }
+ }
+ return true;
+ }
+
+ if (*val == 0 ||
+ pg_strcasecmp(val, "false") == 0 ||
+ pg_strcasecmp(val, "no") == 0 ||
+ pg_strcasecmp(val, "off") == 0 ||
+ pg_strcasecmp(val, "0") == 0)
+ {
+ /* Compression is disabled */
+ return true;
+ }
+
+
+ return zpq_deserialize_compressors(val, compressors, n_compressors);
+}
+
+bool
+zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int selected_alg_mask = 0; /* bitmask of already selected
+ * algorithms to avoid duplicates in
+ * compressors */
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *c_string_dup = strdup(c_string); /* following parsing can
+ * modify the string */
+ char *p = c_string_dup;
+
+ *n_compressors = 0;
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+
+ while (*p != '\0')
+ {
+ char *sep = strchr(p, ',');
+ char *col;
+ int compression_level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ bool found;
+
+ if (sep != NULL)
+ *sep = '\0';
+
+ col = strchr(p, ':');
+ if (col != NULL)
+ {
+ *col = '\0';
+ if (sscanf(col + 1, "%d", &compression_level) != 1)
+ {
+ pg_log_warning("invalid compression level %s in compression option '%s'", col + 1, p);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+ }
+ found = false;
+ for (int i = 0; supported_algorithms[i] != NULL; i++)
+ {
+ if (pg_strcasecmp(p, supported_algorithms[i]) == 0)
+ {
+ if (selected_alg_mask & (1 << i))
+ {
+ /* duplicates are not allowed */
+ pg_log_warning("duplicate algorithm %s in compressors string %s", p, c_string);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+
+ (*compressors)[*n_compressors].impl = i;
+ (*compressors)[*n_compressors].level = compression_level;
+
+ selected_alg_mask |= 1 << i;
+ *n_compressors += 1;
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ pg_log_warning("algorithm %s is not supported", p);
+ }
+ if (sep)
+ p = sep + 1;
+ else
+ break;
+ }
+
+ if (*n_compressors == 0)
+ {
+ free(*compressors);
+ *compressors = NULL;
+ }
+ free(c_string_dup);
+ return true;
+}
+
+char *
+zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors)
+{
+ char *res;
+ char *p;
+ size_t i;
+ size_t total_len = 0;
+ char **supported_algorithms = zs_get_supported_algorithms();
+
+ if (n_compressors == 0)
+ {
+ return NULL;
+ }
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ size_t level_len;
+
+ if (!zs_is_valid_impl_id(compressors[i].impl))
+ {
+ pg_log_warning("algorithm impl_id %d is incorrect", compressors[i].impl);
+ return NULL;
+ }
+
+ /* determine the length of the compression level string */
+ level_len = compressors[i].level == 0 ? 1 : (int) floor(log10(abs(compressors[i].level))) + 1;
+ if (compressors[i].level < 0)
+ {
+ level_len += 1; /* add the leading "-" */
+ }
+
+ /*
+ * single entry looks like "alg_name:compression_level," so +2 is for
+ * ":" and "," symbols
+ */
+ total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
+ }
+
+ res = p = malloc(total_len);
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ p += sprintf(p, "%s:%d,", supported_algorithms[compressors[i].impl], compressors[i].level);
+ }
+ p[-1] = '\0';
+ return res;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4d992dc224..aac90484d9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5334,9 +5334,9 @@
proname => 'pg_stat_get_activity', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'int4',
- proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8}',
- proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id}',
+ proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
prosrc => 'pg_stat_get_activity' },
{ oid => '3318',
descr => 'statistics: information about progress of backends running maintenance command',
@@ -5644,6 +5644,14 @@
proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,wal_write,wal_sync,wal_write_time,wal_sync_time,stats_reset}',
prosrc => 'pg_stat_get_wal' },
+{ oid => '9598', descr => 'statistics: information about network traffic',
+ proname => 'pg_stat_get_network_traffic', proisstrict => 'f', provolatile => 's',
+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o}',
+ proargnames => '{_beid,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
+ prosrc => 'pg_stat_get_network_traffic' },
+
{ oid => '2306', descr => 'statistics: information about SLRU caches',
proname => 'pg_stat_get_slru', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
@@ -5785,6 +5793,10 @@
proname => 'pg_tablespace_location', provolatile => 's', prorettype => 'text',
proargtypes => 'oid', prosrc => 'pg_tablespace_location' },
+{ oid => '9257', descr => 'connection compression algorithm',
+ proname => 'pg_compression_algorithm', provolatile => 's', prorettype => 'text',
+ proargtypes => '', prosrc => 'pg_compression_algorithm' },
+
{ oid => '1946',
descr => 'convert bytea value into some ascii-only text string',
proname => 'encode', prorettype => 'text', proargtypes => 'bytea text',
diff --git a/src/include/common/z_stream.h b/src/include/common/z_stream.h
new file mode 100644
index 0000000000..8229035978
--- /dev/null
+++ b/src/include/common/z_stream.h
@@ -0,0 +1,107 @@
+/*
+ * z_stream.h
+ * Streaming compression
+ */
+
+
+#ifndef Z_STREAM_H
+#define Z_STREAM_H
+
+#include <stdlib.h>
+
+#define ZS_OK (0)
+#define ZS_IO_ERROR (-1)
+#define ZS_DECOMPRESS_ERROR (-2)
+#define ZS_COMPRESS_ERROR (-3)
+#define ZS_STREAM_END (-4)
+#define ZS_DATA_PENDING (-5)
+#define ZS_INCOMPLETE_SRC (-6) /* cannot decompress unless full src message is fetched */
+#define ZS_EXPAND_DST (-7) /* cannot compress: dst buffer is too small for given src size */
+
+struct ZStream;
+typedef struct ZStream ZStream;
+
+#endif
+
+/*
+ * Create compression stream for sending compressed data.
+ * c_alg_impl: index of chosen compression algorithm
+ * c_level: compression c_level
+ */
+extern ZStream * zs_create_compressor(unsigned int c_alg_impl, int c_level);
+
+/*
+ * Create decompression stream for reading compressed data.
+ * d_alg_impl: index of chosen decompression algorithm
+ */
+extern ZStream * zs_create_decompressor(unsigned int d_alg_impl);
+
+/*
+ * Read up to "size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZS_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Write up to "size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZS_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *processed.
+ */
+extern ssize_t zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zs_decompress_error(ZStream * zs);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zs_compress_error(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal rx decompression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal tx compression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * End the compression stream.
+ */
+extern ssize_t zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Free stream created by zs_create_compressor function.
+ */
+extern void zs_compressor_free(ZStream * zs);
+
+/*
+ * Free stream created by zs_create_decompressor function.
+ */
+extern void zs_decompressor_free(ZStream * zs);
+
+/*
+ * Get the name of chosen compression algorithm.
+ */
+extern char const *zs_compress_algorithm_name(ZStream * zs);
+
+/*
+ * Get the name of chosen decompression algorithm.
+ */
+extern char const *zs_decompress_algorithm_name(ZStream * zs);
+
+/*
+ Returns zero terminated array with compression algorithms names
+*/
+extern char **zs_get_supported_algorithms(void);
+
+/*
+ Returns true if provided id is a valid compression algorithm id, otherwise returns false
+*/
+extern bool zs_is_valid_impl_id(unsigned int id);
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000000..fe9eaa0f95
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,117 @@
+/*
+ * zpq_stream.h
+ * Streaming compression for libpq
+ */
+#include "z_stream.h"
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#define ZPQ_DEFAULT_COMPRESSION_LEVEL (1)
+#define ZPQ_INCOMPLETE_HEADER (-6)
+#define ZPQ_FATAL_ERROR (-7)
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t (*zpq_tx_func) (void *arg, void const *data, size_t size);
+typedef ssize_t (*zpq_rx_func) (void *arg, void *data, size_t size);
+
+/*
+ * Descriptor of compression algorithm chosen by client
+ */
+typedef struct zpq_compressor
+{
+ unsigned int impl; /* compression algorithm index */
+ int level; /* compression level */
+} zpq_compressor;
+
+#endif
+
+/*
+ * Create compression stream with rx/tx function for reading/sending compressed data.
+ * tx_func: function for writing compressed data in underlying stream
+ * rx_func: function for reading compressed data from underlying stream
+ * arg: context passed to the function
+ * rx_data: received data (compressed data already fetched from input stream)
+ * rx_data_size: size of data fetched from input stream
+ */
+extern ZpqStream * zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size);
+
+/*
+ * Write up to "src_size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZPQ_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *src_processed.
+ */
+extern ssize_t zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed);
+
+/*
+ * Read up to "dst_size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZPQ_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zpq_read(ZpqStream * zpq, void *dst, size_t dst_size);
+
+/*
+ * Return true if non-flushed data left in internal rx decompression buffer.
+ */
+extern bool zpq_buffered_rx(ZpqStream * zpq);
+
+/*
+ * Return true if non-flushed data left in internal tx compression buffer.
+ */
+extern bool zpq_buffered_tx(ZpqStream * zpq);
+
+/*
+ * Free stream created by zs_create function.
+ */
+extern void zpq_free(ZpqStream * zpq);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zpq_decompress_error(ZpqStream * zpq);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zpq_compress_error(ZpqStream * zpq);
+
+/*
+ * Get the name of the current compression algorithm.
+ */
+extern char const *zpq_compress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Get the name of the current decompression algorithm.
+ */
+extern char const *zpq_decompress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Parse comma-separated list of compression algorithms. It can be either explicitly specified, or include all algorithms supported by client library.
+ * Returns:
+ * - true if the compression string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or if compression is disabled, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+extern bool
+ zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Serialize the compressors array to string so it can be transmitted to the other side during the compression startup.
+ * For example, for array of two compressors (zstd, level 1), (zlib, level 2) resulting string would look like "zstd:1,zlib:2".
+ * Returns the resulting string.
+ */
+extern char
+ *zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors);
+
+/* Deserialize the compressors string received during the compression setup to a compressors array.
+ * For example, for string "zstd:1,zlib:2" compressors would be populated with 2 elements: (zstd, level 1), (zlib, level 2).
+ * Returns:
+ * - true if the compressors string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or string is empty, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+bool
+ zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors);
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 02015efe13..bc3e47b51f 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -188,6 +188,9 @@ typedef struct Port
int keepalives_count;
int tcp_user_timeout;
+ char *compression_algorithms; /* Compression algorithms supported by
+ * client */
+
/*
* GSSAPI structures.
*/
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 6b67a2a318..794dbb4911 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -71,6 +71,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port *port);
extern int pq_getbytes(char *s, size_t len);
extern void pq_startmsgread(void);
extern void pq_endmsgread(void);
diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h
index 1bcc189dee..c46f5e00d9 100644
--- a/src/include/libpq/pqcomm.h
+++ b/src/include/libpq/pqcomm.h
@@ -137,6 +137,9 @@ typedef uint32 PacketLen;
extern bool Db_user_namespace;
+/* List of allowed compression algorithms */
+extern char *libpq_compress_algorithms;
+
/*
* In protocol 3.0 and later, the startup packet length is not fixed, but
* we set an arbitrary limit on it anyway. This is just to prevent simple
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 8042b817df..6bbf7d530e 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -144,6 +144,12 @@ typedef struct PgBackendStatus
/* application name; MUST be null-terminated */
char *st_appname;
+ /* client-server traffic information */
+ uint64 st_rx_raw_bytes;
+ uint64 st_tx_raw_bytes;
+ uint64 st_rx_compressed_bytes;
+ uint64 st_tx_compressed_bytes;
+
/*
* Current command string; MUST be null-terminated. Note that this string
* possibly is truncated in the middle of a multi-byte character. As
@@ -301,6 +307,7 @@ extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
+extern void pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes);
extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
int buflen);
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 40a96c881d..21c29c0acc 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,20 @@ endif
# The MSVC build system scrapes OBJS from this file. If you change any of
# the conditional additions of files to OBJS, update Mkvcbuild.pm to match.
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+SHLIB_LINK += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
+# We can't use Makefile variables here because the MSVC build system scrapes
+# OBJS from this file.
+
+
OBJS = \
$(WIN32RES) \
fe-auth-scram.o \
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index e8bcc88370..33ce6f2513 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -186,3 +186,6 @@ PQpipelineStatus 183
PQsetTraceFlags 184
PQmblenBounded 185
PQsendFlushRequest 186
+PQcompressor 187
+PQdecompressor 188
+PQreadPending 189
\ No newline at end of file
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 9b6a6939f0..e121d4e71a 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -24,6 +24,7 @@
#include "common/ip.h"
#include "common/link-canary.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "common/string.h"
#include "fe-auth.h"
#include "libpq-fe.h"
@@ -339,6 +340,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "PGCOMPRESSION", NULL, NULL,
+ "Libpq-compression", "", 16,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 15, /* sizeof("prefer-standby") = 15 */
@@ -447,6 +452,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zpqStream);
+ conn->zpqStream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -3271,11 +3280,14 @@ keep_going: /* We will come back to here until there is
}
/*
- * Validate message type: we expect only an authentication
- * request or an error here. Anything else probably means
- * it's not Postgres on the other end at all.
+ * Validate message type. We expect only:
+ * - authentication request ('R')
+ * - error ('E')
+ * - protocol compression acknowledgment ('z')
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
{
appendPQExpBuffer(&conn->errorMessage,
libpq_gettext("expected authentication request from server, but received %c\n"),
@@ -3353,6 +3365,60 @@ keep_going: /* We will come back to here until there is
return PGRES_POLLING_READING;
}
+ if (beresp == 'z') /* Switch on compression */
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ char *resp = malloc(msgLength);
+
+ pqGetnchar(resp, msgLength, conn);
+
+ if (!zpq_deserialize_compressors(resp, &compressors, &n_compressors))
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server returned unrecognized compression setting: %s\n"),
+ resp);
+ free(resp);
+ goto error_return;
+ }
+ free(resp);
+
+ if (n_compressors == 0)
+ {
+ /*
+ * If there are no compressors returned, it means that
+ * the server rejected all sent compression
+ * algorithms. Mark the data as consumed and do not
+ * enable the compression.
+ */
+ conn->inStart = conn->inCursor;
+ goto keep_going;
+ }
+
+ Assert(!conn->zpqStream);
+ conn->zpqStream = zpq_create(compressors, n_compressors,
+ (zpq_tx_func) pqsecure_write, (zpq_rx_func) pqsecure_read,
+ conn,
+ &conn->inBuffer[conn->inCursor],
+ conn->inEnd - conn->inCursor);
+ if (!conn->zpqStream)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("failed to initialize compression\n"));
+ free(compressors);
+ goto error_return;
+ }
+ /* reset buffer */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+
+ if (conn->n_compressors != 0 && beresp == 'v') /* NegotiateProtocolVersion */
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server does not support protocol compression\n"));
+ goto error_return;
+ }
+
/* Handle errors. */
if (beresp == 'E')
{
@@ -4097,6 +4163,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
@@ -6628,6 +6696,22 @@ PQuser(const PGconn *conn)
return conn->pguser;
}
+char *
+PQcompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_compress_algorithm_name(conn->zpqStream);
+}
+
+char *
+PQdecompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_decompress_algorithm_name(conn->zpqStream);
+}
+
char *
PQpass(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 6c7b3df012..2061862688 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2010,9 +2010,7 @@ PQgetResult(PGconn *conn)
* EOF indication. We expect therefore that this won't result in any
* undue delay in reporting a previous write failure.)
*/
- if (flushResult ||
- pqWait(true, false, conn) ||
- pqReadData(conn) < 0)
+ if (flushResult || pqWait(true, false, conn) || pqReadData(conn) < 0)
{
/*
* conn->errorMessage has been set by pqWait or pqReadData. We
@@ -3785,6 +3783,12 @@ pqPipelineFlush(PGconn *conn)
return 0;
}
+int
+PQreadPending(PGconn *conn)
+{
+ return pqReadPending(conn);
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 9a2a970293..8b7ab3367e 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,12 +53,24 @@
#include "pg_config_paths.h"
#include "port/pg_bswap.h"
+#include <common/zpq_stream.h>
+
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
+/*
+ * Use zpq_read if compression is switched on
+ */
+#define pq_read_conn(conn) \
+ (conn->zpqStream \
+ ? zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd) \
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd))
+
/*
* PQlibVersion: return the libpq version number
*/
@@ -617,10 +629,17 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -712,10 +731,18 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
+
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -826,12 +853,18 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered_tx(conn->zpqStream))
{
int sent;
+ size_t processed = 0;
+ /*
+ * Use zpq_write if compression is switched on
+ */
+ sent = conn->zpqStream
+ ? zpq_write(conn->zpqStream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -839,8 +872,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+: pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +932,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered_tx(conn->zpqStream))
{
/*
* We didn't send it all, wait till we can send more.
@@ -989,6 +1025,8 @@ pqFlush(PGconn *conn)
int
pqWait(int forRead, int forWrite, PGconn *conn)
{
+ if (forRead && conn->inCursor < conn->inEnd)
+ return 0;
return pqWaitTimed(forRead, forWrite, conn, (time_t) -1);
}
@@ -1046,6 +1084,9 @@ pqWriteReady(PGconn *conn)
*
* If SSL is in use, the SSL buffer is checked prior to checking the socket
* for read data directly.
+ *
+ * If ZPQ stream is in use, the ZPQ buffer is checked prior to checking
+ * the socket for read data directly.
*/
static int
pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
@@ -1061,14 +1102,10 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return -1;
}
-#ifdef USE_SSL
- /* Check for SSL library buffering read bytes */
- if (forRead && conn->ssl_in_use && pgtls_read_pending(conn))
+ if (forRead && (pqReadPending(conn) > 0))
{
- /* short-circuit the select */
return 1;
}
-#endif
/* We will retry as long as we get EINTR */
do
@@ -1088,6 +1125,33 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return result;
}
+/*
+ * Check if there is some data pending in ZPQ / SSL read buffers.
+ * Returns -1 on failure, 0 if no, 1 if yes.
+ */
+int
+pqReadPending(PGconn *conn)
+{
+ if (!conn)
+ return -1;
+
+ /* check for ZPQ stream buffered read bytes */
+ if (zpq_buffered_rx(conn->zpqStream))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+
+#ifdef USE_SSL
+ /* Check for SSL library buffering read bytes */
+ if (conn->ssl_in_use && pgtls_read_pending(conn))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+#endif
+ return 0;
+}
/*
* Check a file descriptor for read and/or write data, possibly waiting.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 9ab3bf1fcb..0619e1ab0d 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2174,6 +2174,50 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
return startpacket;
}
+/*
+ * Build comma-separated list of compression algorithms requested by client.
+ * It can be either explicitly specified by user in connection string, or
+ * include all algorithms supported by client library.
+ * This function returns true if the compression string is successfully parsed and
+ * stores a comma-separated list of algorithms in *client_compressors.
+ * If compression is disabled, then NULL is assigned to *client_compressors.
+ * Also it creates an array of compressor descriptors, each element of which corresponds to
+ * the corresponding algorithm name in *client_compressors list. This array is stored in PGconn
+ * and is used during handshake when a compression acknowledgment response is received from the server.
+ */
+static bool
+build_compressors_list(PGconn *conn, char **client_compressors, bool build_descriptors)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors))
+ {
+ return false;
+ }
+
+ *client_compressors = NULL;
+ if (build_descriptors)
+ {
+ conn->compressors = compressors;
+ conn->n_compressors = n_compressors;
+ }
+
+ if (n_compressors == 0)
+ {
+ /* no compressors available, return */
+ return true;
+ }
+
+ *client_compressors = zpq_serialize_compressors(compressors, n_compressors);
+
+ if (!build_descriptors)
+ {
+ free(compressors);
+ }
+ return true;
+}
+
/*
* Build a startup packet given a filled-in PGconn structure.
*
@@ -2220,6 +2264,18 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("replication", conn->replication);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
+ if (conn->compression && conn->compression[0])
+ {
+ char *client_compression_algorithms;
+
+ if (build_compressors_list((PGconn *) conn, &client_compression_algorithms, packet == NULL))
+ {
+ if (client_compression_algorithms)
+ {
+ ADD_STARTUP_OPTION("_pq_.compression", client_compression_algorithms);
+ }
+ }
+ }
if (conn->send_appname)
{
/* Use appname if present, otherwise use fallback */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index a6fd69aceb..1e951320d8 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -340,6 +340,8 @@ extern char *PQhostaddr(const PGconn *conn);
extern char *PQport(const PGconn *conn);
extern char *PQtty(const PGconn *conn);
extern char *PQoptions(const PGconn *conn);
+extern char *PQcompressor(const PGconn *conn);
+extern char *PQdecompressor(const PGconn *conn);
extern ConnStatusType PQstatus(const PGconn *conn);
extern PGTransactionStatusType PQtransactionStatus(const PGconn *conn);
extern const char *PQparameterStatus(const PGconn *conn,
@@ -498,6 +500,9 @@ extern PGPing PQpingParams(const char *const *keywords,
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
+extern int
+ PQreadPending(PGconn *conn);
+
/*
* "Fast path" interface --- not really recommended for application
* use
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 490458adef..1e54a4c79f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "fe-auth-sasl.h"
#include "pqexpbuffer.h"
@@ -340,6 +341,7 @@ typedef struct pg_conn_host
* found in password file. */
} pg_conn_host;
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -392,6 +394,14 @@ struct pg_conn
* "sspi") */
char *ssl_min_protocol_version; /* minimum TLS protocol version */
char *ssl_max_protocol_version; /* maximum TLS protocol version */
+
+ char *compression; /* stream compression (boolean value, "any" or
+ * list of compression algorithms separated by
+ * comma) */
+ zpq_compressor *compressors; /* descriptors of compression algorithms
+ * chosen by client */
+ unsigned n_compressors; /* size of compressors array */
+
char *target_session_attrs; /* desired session properties */
/* Optional file to write trace info to */
@@ -572,6 +582,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream *zpqStream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
@@ -697,6 +710,7 @@ extern int pqWaitTimed(int forRead, int forWrite, PGconn *conn,
time_t finish_time);
extern int pqReadReady(PGconn *conn);
extern int pqWriteReady(PGconn *conn);
+extern int pqReadPending(PGconn *conn);
/* === in fe-secure.c === */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index b58b062b10..e4363bbf7c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1765,7 +1765,7 @@ pg_stat_activity| SELECT s.datid,
s.query_id,
s.query,
s.backend_type
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_all_indexes| SELECT c.oid AS relid,
@@ -1877,8 +1877,14 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_auth AS gss_authenticated,
s.gss_princ AS principal,
s.gss_enc AS encrypted
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
+pg_stat_network_traffic| SELECT s.pid,
+ s.rx_raw_bytes,
+ s.tx_raw_bytes,
+ s.rx_compressed_bytes,
+ s.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
@@ -2047,7 +2053,7 @@ pg_stat_replication| SELECT s.pid,
w.sync_priority,
w.sync_state,
w.reply_time
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_replication_slots| SELECT s.slot_name,
@@ -2081,7 +2087,7 @@ pg_stat_ssl| SELECT s.pid,
s.ssl_client_dn AS client_dn,
s.ssl_client_serial AS client_serial,
s.ssl_issuer_dn AS issuer_dn
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 404c45a6f3..c41370cd28 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -130,7 +130,7 @@ sub mkvcbuild
keywords.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c
rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c
- username.c wait_error.c wchar.c);
+ username.c wait_error.c wchar.c z_stream.c zpq_stream.c);
if ($solution->{options}->{openssl})
{
--
2.30.2
=
0002-Add-ZSTD-support.patch (application/octet-stream, 17.8K) download
From 0c3bc076dc5c26a801df13e6a535c8626f7e10a8 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:59:01 +0500
Subject: [PATCH 2/2] Add ZSTD support
---
configure | 214 +++++++++++++++++++++++++++++++
configure.ac | 36 ++++++
src/Makefile.global.in | 1 +
src/backend/Makefile | 4 +
src/common/z_stream.c | 228 ++++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
src/interfaces/libpq/Makefile | 5 +
src/tools/msvc/Solution.pm | 1 +
8 files changed, 492 insertions(+)
diff --git a/configure b/configure
index 3b19105328..e49a16f8f4 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,9 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -868,6 +871,7 @@ with_libxslt
with_system_tzdata
with_zlib
with_lz4
+with_zstd
with_gnu_ld
with_ssl
with_openssl
@@ -897,6 +901,8 @@ XML2_CFLAGS
XML2_LIBS
LZ4_CFLAGS
LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
LDFLAGS_EX
LDFLAGS_SL
PERL
@@ -1576,6 +1582,7 @@ Optional Packages:
use system time zone data in DIR
--without-zlib do not use Zlib
--with-lz4 build with LZ4 support
+ --with-zstd build with ZSTD support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -1605,6 +1612,8 @@ Some influential environment variables:
XML2_LIBS linker flags for XML2, overriding pkg-config
LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config
LZ4_LIBS linker flags for LZ4, overriding pkg-config
+ ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+ ZSTD_LIBS linker flags for ZSTD, overriding pkg-config
LDFLAGS_EX extra linker flags for linking executables only
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
@@ -9033,6 +9042,145 @@ fi
done
fi
+#
+# Zstd
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+
+$as_echo "#define HAVE_LIBZSTD 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd >= 1.0.0" >&5
+$as_echo_n "checking for libzstd >= 1.0.0... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+ pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+ pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ else
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$ZSTD_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (libzstd >= 1.0.0) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+ ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -12523,6 +12671,58 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_decompressStream in -lzstd" >&5
+$as_echo_n "checking for ZSTD_decompressStream in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_decompressStream+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_decompressStream ();
+int
+main ()
+{
+return ZSTD_decompressStream ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_decompressStream=yes
+else
+ ac_cv_lib_zstd_ZSTD_decompressStream=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_decompressStream" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_decompressStream" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_decompressStream" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13854,6 +14054,20 @@ fi
done
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index e77d4dcf2d..3d554869ad 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,28 @@ if test "$with_lz4" = yes; then
done
fi
+#
+# Zstd
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support], [AC_DEFINE([HAVE_LIBZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+ PKG_CHECK_MODULES(ZSTD, [libzstd >= 1.0.0])
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -1242,6 +1264,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_decompressStream, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1487,6 +1516,13 @@ if test "$with_lz4" = yes; then
AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 79ed589ce9..71e14ee68f 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -197,6 +197,7 @@ with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
with_lz4 = @with_lz4@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index f65f16fc48..2074d23d5d 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,10 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
ifeq ($(with_lz4),yes)
LIBS += -llz4
endif
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index d94914e7dd..6f490ebe78 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -89,6 +89,231 @@ struct ZStream
bool not_flushed;
};
+#if HAVE_LIBZSTD
+
+#include <stdlib.h>
+#include <zstd.h>
+
+/*
+ * Maximum allowed back-reference distance, expressed as power of 2.
+ * This setting controls max compressor/decompressor window size.
+ * More details https://github.com/facebook/zstd/blob/v1.4.7/lib/zstd.h#L536
+ */
+#define ZSTD_WINDOWLOG_LIMIT 23 /* set max window size to 8MB */
+
+
+typedef struct ZS_ZSTD_CStream
+{
+ ZSTD_CStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_CStream;
+
+typedef struct ZS_ZSTD_DStream
+{
+ ZSTD_DStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_DStream;
+
+static void *
+zstd_create_compressor(int level)
+{
+ size_t rc;
+ ZS_ZSTD_CStream *c_stream = (ZS_ZSTD_CStream *) malloc(sizeof(ZS_ZSTD_CStream));
+
+ c_stream->stream = ZSTD_createCStream();
+ rc = ZSTD_initCStream(c_stream->stream, level);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeCStream(c_stream->stream);
+ free(c_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_CCtx_setParameter(c_stream->stream, ZSTD_c_windowLog, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ c_stream->error = NULL;
+ return c_stream;
+}
+
+static void *
+zstd_create_decompressor()
+{
+ size_t rc;
+ ZS_ZSTD_DStream *d_stream = (ZS_ZSTD_DStream *) malloc(sizeof(ZS_ZSTD_DStream));
+
+ d_stream->stream = ZSTD_createDStream();
+ rc = ZSTD_initDStream(d_stream->stream);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeDStream(d_stream->stream);
+ free(d_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_DCtx_setParameter(d_stream->stream, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ d_stream->error = NULL;
+ return d_stream;
+}
+
+static ssize_t
+zstd_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+ size_t rc;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ rc = ZSTD_decompressStream(ds->stream, &out, &in);
+
+ *src_processed = in.pos;
+ *dst_processed = out.pos;
+ if (ZSTD_isError(rc))
+ {
+ ds->error = ZSTD_getErrorName(rc);
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ if (rc == 0)
+ {
+ return ZS_STREAM_END;
+ }
+
+ if (out.pos == out.size)
+ {
+ /*
+ * if `output.pos == output.size`, there might be some data left
+ * within internal buffers
+ */
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ if (in.pos < src_size) /* Has something to compress in input buffer */
+ {
+ size_t rc = ZSTD_compressStream(cs->stream, &out, &in);
+
+ *dst_processed = out.pos;
+ *src_processed = in.pos;
+ if (ZSTD_isError(rc))
+ {
+ cs->error = ZSTD_getErrorName(rc);
+ return ZS_COMPRESS_ERROR;
+ }
+ }
+
+ if (in.pos == src_size) /* All data is compressed: flush internal zstd
+ * buffer */
+ {
+ size_t tx_not_flushed = ZSTD_flushStream(cs->stream, &out);
+
+ *dst_processed = out.pos;
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ size_t tx_not_flushed;
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_outBuffer output;
+
+ output.dst = dst;
+ output.pos = 0;
+ output.size = dst_size;
+
+ do
+ {
+ tx_not_flushed = ZSTD_endStream(cs->stream, &output);
+ } while ((tx_not_flushed > 0) && (output.pos < output.size));
+
+ *dst_processed = output.pos;
+
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static void
+zstd_free_compressor(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ ZSTD_freeCStream(cs->stream);
+ free(cs);
+ }
+}
+
+static void
+zstd_free_decompressor(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ ZSTD_freeDStream(ds->stream);
+ free(ds);
+ }
+}
+
+static char const *
+zstd_compress_error(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ return cs->error;
+}
+
+static char const *
+zstd_decompress_error(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ return ds->error;
+}
+
+static char const *
+zstd_name(void)
+{
+ return "zstd";
+}
+
+#endif
+
#if HAVE_LIBZ
#include <stdlib.h>
@@ -421,6 +646,9 @@ no_compression_name(void)
*/
static ZAlgorithm const zs_algorithms[] =
{
+#if HAVE_LIBZSTD
+ {zstd_name, zstd_create_compressor, zstd_create_decompressor, zstd_decompress, zstd_compress, zstd_free_compressor, zstd_free_decompressor, zstd_compress_error, zstd_decompress_error, zstd_end},
+#endif
#if HAVE_LIBZ
{zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
#endif
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 7525c16597..a63685a639 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 21c29c0acc..85fda6b236 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -34,6 +34,11 @@ LIBS += -llz4
SHLIB_LINK += -llz4
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
ifeq ($(with_zlib),yes)
LIBS += -lz
SHLIB_LINK += -lz
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 2c8cd521e9..5ecd0a1832 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
--
2.30.2
=
* Re: libpq compression (part 2)
@ 2022-01-01 17:25 Justin Pryzby <pryzby@telsasoft.com>
parent: Daniil Zakhlystov <usernamedt@yandex-team.ru>
1 sibling, 2 replies; 27+ messages in thread
From: Justin Pryzby @ 2022-01-01 17:25 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; +Cc: pgsql-hackers@lists.postgresql.org
Thanks for working on this. The patch looks to be in good shape - I hope more
people will help to review and test it. I took the liberty of creating a new
CF entry. http://cfbot.cputube.org/daniil-zakhlystov.html
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
I think this is backwards , and should say != -1 ?
As written, the server GUC libpq_compression defaults to "on", and the client
doesn't request compression. I think the server GUC should default to off.
I failed to convince Kontantin about this last year. The reason is that 1)
it's a new feature; 2) with security implications. An admin should need to
"opt in" to this. I still wonder if this should be controlled by a new "TYPE"
in pg_hba (rather than a GUC); that would make it exclusive of SSL.
However, I also think that in the development patches both client and server
should enable compression by default, to allow it to be exercized by cfbot.
For the other compression patches I worked on, I had an 0001 commit where the
feature default was off, plus following patches which enabled the feature by
default - only for cfbot and not intended for commit.
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms;
=> The global variable is conventionally initialized to the same default as
guc.c (even though the GUC default value will be applied at startup).
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
=> I think NegotiateProtocolVersion and 'v' are from an old version of the
patch and no longer used ?
There's a handful of compiler warnings:
| z_stream.c:311:5: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
+ ereport(LOG, (errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms)));
+ return 0;
=> Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
+ * Report current transaction start timestamp as the specified value.
+ * Zero means there is no active transaction.
=> The comment doesn't correspond to the function (copy+paste).
+ return id >= 0 && id < (sizeof(zs_algorithms) / sizeof(*zs_algorithms));
+ size_t n_algorithms = sizeof(zs_algorithms) / sizeof(*zs_algorithms);
=> You can use lengthof() for these.
Maybe pg_stat_network_traffic should show the compression?
It'd have to show the current client-server and server-client values.
Or maybe that's not useful since it can change dynamically (unless it were
reset when the compression method was changed).
I think the psql conninfo Compressor/Decompressor line may be confusing.
Maybe it should talk about Client->Server and Server->Client.
Maybe it should be displayed on a single line.
Actually, right now the \conninfo part is hardly useful, since the compressor
is allocated lazily/"on demand", so it shows the compression of some previous
command, but maybe not the last command, and maybe not the next command...
I'm not sure it's needed to allow the compression to be changed dynamically,
and the generality to support a different compression mthod for each libpq
message type seems excessive. Maybe it's enough to check that the message type
is one of VALID_LONG_MESSAGE_TYPE and its length is long enough.
I wonder whether the asymmetric compression idea is useful. The only
application I can see for this is that a server might allow to *decompress*
data from a client, but may not want to *compress* data to a client.
What happenes if the compression doesn't decrease the message size ? I don't
see anything that allows sending the original, raw data. The advantage would
be that the remote side doesn't incur the overhead of decompression.
I hit this assertion with PGCOMPRESSION=lz4 (but not zlib), and haven't yet
found the cause.
TRAP: FailedAssertion("decBytes > 0", File: "z_stream.c", Line: 315, PID: 21180)
postgres: pryzbyj postgres 127.0.0.1(46154) idle(ExceptionalCondition+0x99)[0x5642137806d9]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(+0x632bfe)[0x5642137d4bfe]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(zs_read+0x2b)[0x5642137d4ebb]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(zpq_read+0x192)[0x5642137d1172]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(+0x378b8a)[0x56421351ab8a]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(pq_getbyte+0x1f)[0x56421351c12f]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(PostgresMain+0xf96)[0x56421365fcc6]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(+0x428b69)[0x5642135cab69]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(PostmasterMain+0xd1c)[0x5642135cbbac]
postgres: pryzbyj postgres 127.0.0.1(46154) idle(main+0x220)[0x5642132f9a40]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fb70ef45b97]
Maybe you should reset the streams between each compression message (even if
it's using the same compression algorithm). This might allow better
compression. You could either unconditionally call zs_compressor_free()/
zs_create_compressor(). Or, add a new, optional interface for resetting the
stream (and if that isn't set for a given compression method, then free+create
the stream). For LZ4, that'd be LZ4_resetStream_fast(), but only for
v1.9.0+...
Some minor formatting issues:
- There are spaces rather than tabs in a few files; you can maybe fix it by
piping the file through unexpand -t4.
- pointers in function declarations should have no space after the stars:
+zs_buffered(ZStream * zs)
- There's a few extraneous whitespace changes:
ConfigureNamesEnum, pq_buffer_has_data, libpq-int.h.
- Some lines are too long:
git log -2 -U1 '*.[ch]' |grep -E '.{80}|^diff' |less
- Some 1-line "if" statements would be easier to read without braces {}.
- Some multi-line comments should have "*/" on a separate line:
git log -3 -p |grep -E '^\+ \*.*\*\/'
git log -3 -p '*.c' |grep -E '^\+[^/]*\*.*\*\/'
--
Justin
Attachments: download all (8 files, .tar.gz)
0001-Implement-libpq-compression.patch (text/x-diff, 114.5K) download
From e1bbb50ee548cd5d0246dfa0d4bfad93a46c0328 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:58:38 +0500
Subject: [PATCH 01/10] Implement libpq compression
---
.../postgres_fdw/expected/postgres_fdw.out | 2 +-
doc/src/sgml/config.sgml | 21 +
doc/src/sgml/libpq.sgml | 37 +
doc/src/sgml/protocol.sgml | 174 +++
src/Makefile.global.in | 1 +
src/backend/Makefile | 8 +
src/backend/catalog/system_views.sql | 9 +
src/backend/libpq/pqcomm.c | 277 ++++-
src/backend/postmaster/postmaster.c | 10 +
src/backend/utils/activity/backend_status.c | 30 +
src/backend/utils/adt/pgstatfuncs.c | 50 +-
src/backend/utils/misc/guc.c | 30 +
src/bin/pgbench/pgbench.c | 17 +-
src/bin/psql/command.c | 23 +
src/common/Makefile | 4 +-
src/common/z_stream.c | 659 +++++++++++
src/common/zpq_stream.c | 1001 +++++++++++++++++
src/include/catalog/pg_proc.dat | 18 +-
src/include/common/z_stream.h | 107 ++
src/include/common/zpq_stream.h | 117 ++
src/include/libpq/libpq-be.h | 3 +
src/include/libpq/libpq.h | 1 +
src/include/libpq/pqcomm.h | 3 +
src/include/utils/backend_status.h | 7 +
src/interfaces/libpq/Makefile | 14 +
src/interfaces/libpq/exports.txt | 3 +
src/interfaces/libpq/fe-connect.c | 92 +-
src/interfaces/libpq/fe-exec.c | 10 +-
src/interfaces/libpq/fe-misc.c | 90 +-
src/interfaces/libpq/fe-protocol3.c | 56 +
src/interfaces/libpq/libpq-fe.h | 5 +
src/interfaces/libpq/libpq-int.h | 14 +
src/test/regress/expected/rules.out | 14 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
34 files changed, 2820 insertions(+), 89 deletions(-)
create mode 100644 src/common/z_stream.c
create mode 100644 src/common/zpq_stream.c
create mode 100644 src/include/common/z_stream.h
create mode 100644 src/include/common/zpq_stream.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7720ab9c581..5e4f38871be 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9509,7 +9509,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
+HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, compression, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index afbb6c35e30..8840edf9130 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1035,6 +1035,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-libpq-compression" xreflabel="libpq_compression">
+ <term><varname>libpq_compression</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>libpq_compression</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter controls the available client-server traffic compression methods.
+ It allows rejecting compression requests even if it is supported by the server (for example, due to security, or CPU consumption).
+ The default is <literal>on</literal>, which means that all supported compression methods are allowed.
+ For more precise control, a list of the allowed compression methods can be specified.
+ For example, to allow only <literal>lz4</literal> and <literal>zlib</literal>, set the setting to <literal>lz4,zlib</literal>.
+ Also, maximal allowed compression level can be specified for each method, e.g. <literal>lz4:1,zlib:2</literal> setting will set the
+ maximal compression level for <literal>lz4</literal> to 1 and <literal>zlib</literal> to 2.
+ If a client requests the compression with a higher compression level, it will be set to the maximal allowed one.
+ Default (and recommended) maximal compression level for each algorithm is 1.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 14f35d37f66..5a16b4b64bb 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1260,6 +1260,43 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. The client sends a request with
+ a list of compression algorithms. Compression can be requested by a client
+ by including the <literal>"compression"</literal> option in its connection string.
+ This can either be a boolean value to enable or disable compression
+ (<literal>"true"</literal>/<literal>"false"</literal>,
+ <literal>"on"</literal>/<literal>"off"</literal>,
+ <literal>"yes"</literal>/<literal>"no"</literal>,
+ <literal>"1"</literal>/<literal>"0"</literal>),
+ <literal>"any"</literal>,
+ or an explicit list of comma-separated compression algorithms
+ which can optionally include compression level (<literal>"zlib,lz4:2"</literal>).
+ If compression is enabled but an algorithm is not explicitly specified,
+ the client library sends its full list of supported algorithms.
+ The server intersects the received compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ </para>
+ <para>
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ Compressed data is transmitted via the CompressedData messages.
+ </para>
+ <para>
+ Support for compression algorithms must be enabled when the server is compiled.
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option). In both cases, streaming mode is used.
+ By default, compression is not requested by the client.
+ Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 5bc2e7a591a..17a095b8182 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ It is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especially useful for importing/exporting data to/from the database using the <literal>COPY</literal> command
+ and for replication (both physical and logical). Compression can also reduce the server's response time
+ for queries returning a large amount of data (for example, JSON, BLOBs, text, ...).
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option).
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -262,6 +271,21 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CompressionAck</term>
+ <listitem>
+ <para>
+ The server accepts the client's compression request.
+ Compression is requested when a client connection includes the "compression" option, which contains a list of the requested compression algorithms.
+ The server intersects the requested compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>AuthenticationOk</term>
<listitem>
@@ -3577,6 +3601,141 @@ following:
</varlistentry>
+<varlistentry>
+<term>
+CompressionAck (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ List of the negotiated compression algorithms.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
+<varlistentry>
+<term>
+SetCompressionMethod (F & B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('k')
+</term>
+<listitem>
+<para>
+ Switch the current compression algorithm. Following CompressedData messages will be compressed via the compressed algorithm specified in this message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32(5)
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ Index of the new compression algorithm.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+CompressedData (F & B)
+</term>
+<listitem>
+<para>
+
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('m')
+</term>
+<listitem>
+<para>
+ Identifies the message as compressed data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ Compressed message data.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
AuthenticationSASLContinue (B)
@@ -6142,6 +6301,21 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>_pq_.compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. The value is a list of compression algorithms requested by the client with an optional
+ specification of compression level: <literal>"zlib,lz4:5"</literal>.
+ If the server does not support compression, the backend will ignore the _pq_.compression
+ parameter and will not send the CompressionAck message to the frontend.
+ By default, compression is disabled. Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 05c54b27def..79ed589ce94 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_lz4 = @with_lz4@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 0da848b1fd3..f65f16fc480 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 80d73c40ad2..c5731a2d89e 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -844,6 +844,15 @@ CREATE VIEW pg_stat_activity AS
LEFT JOIN pg_database AS D ON (S.datid = D.oid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
+CREATE VIEW pg_stat_network_traffic AS
+ SELECT
+ S.pid,
+ S.rx_raw_bytes,
+ S.tx_raw_bytes,
+ S.rx_compressed_bytes,
+ S.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL) AS S;
+
CREATE VIEW pg_stat_replication AS
SELECT
S.pid,
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 4c37df09cff..bac4d9f14b7 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -60,6 +60,7 @@
#include <signal.h>
#include <fcntl.h>
#include <grp.h>
+#include <pgstat.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/socket.h>
@@ -77,11 +78,14 @@
#include "common/ip.h"
#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
#include "miscadmin.h"
#include "port/pg_bswap.h"
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/builtins.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -108,6 +112,9 @@
int Unix_socket_permissions;
char *Unix_socket_group;
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms;
+
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -130,6 +137,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream * PqStream;
+
+
/*
* Message status
*/
@@ -167,6 +177,145 @@ const PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+static ssize_t
+write_compressed(void *arg, void const *data, size_t size)
+{
+ ssize_t rc = secure_write((Port *) arg, (void *) data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, 0, rc);
+ return rc;
+}
+
+static ssize_t
+read_compressed(void *arg, void *data, size_t size)
+{
+ ssize_t rc = secure_read((Port *) arg, data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, rc, 0);
+ return rc;
+}
+
+/*
+ * Send the chosen compression algorithms to the client
+ */
+static void
+SendCompressionACK(char *compressors_str)
+{
+ StringInfoData buf;
+
+ pq_beginmessage(&buf, 'z');
+ pq_sendstring(&buf, compressors_str);
+ pq_endmessage(&buf);
+ pq_flush();
+}
+
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port *port)
+{
+ zpq_compressor *server_compressors;
+ zpq_compressor *client_compressors;
+ zpq_compressor *res_compressors;
+ size_t n_client_compressors;
+ size_t n_server_compressors;
+ size_t n_res_compressors = 0;
+ char *res_compressors_str;
+ char *client_compression_algorithms = port->compression_algorithms;
+
+ if (!client_compression_algorithms || !libpq_compress_algorithms)
+ {
+ return 0;
+ }
+
+ if (!zpq_parse_compression_setting(libpq_compress_algorithms, &server_compressors, &n_server_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms)));
+ return 0;
+ }
+
+ if (n_server_compressors == 0)
+ {
+ /*
+ * No enabled server compressors available, abort the compression
+ * initialization.
+ */
+ /* Send the compression acknowledgment with empty compressors list. */
+ char *empty_response = "";
+
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ if (!zpq_deserialize_compressors(client_compression_algorithms, &client_compressors, &n_client_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse received client compression methods: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ if (n_client_compressors == 0)
+ {
+ /*
+ * client did not provide any compression algorithms that server is
+ * compiled to support
+ */
+ ereport(LOG, (errmsg("server doesn't support any of the compression methods requested by the client: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ res_compressors = malloc(Max(n_client_compressors, n_server_compressors) * sizeof(zpq_compressor));
+
+ /*
+ * Intersect client and server compressors to determine the final list of
+ * the supported compressors. O(N^2) is negligible because of a small
+ * number of the compression methods.
+ */
+ for (size_t i = 0; i < n_client_compressors; i++)
+ {
+ for (size_t j = 0; j < n_server_compressors; j++)
+ {
+ if (client_compressors[i].impl == server_compressors[j].impl)
+ {
+ res_compressors[n_res_compressors].impl = client_compressors[i].impl;
+ /* prefer the lower compression level */
+ res_compressors[n_res_compressors++].level = Min(client_compressors[i].level, server_compressors[j].level);
+ break;
+ }
+ }
+ }
+ free(client_compressors);
+ free(server_compressors);
+
+ if (n_res_compressors == 0)
+ {
+ char *empty_response = "";
+
+ ereport(LOG, (errmsg("did not find any matches between the client and server compression methods, not enabling compression")));
+ free(res_compressors);
+ /* send the compression ack with empty compressors list */
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ res_compressors_str = zpq_serialize_compressors(res_compressors, n_res_compressors);
+ SendCompressionACK(res_compressors_str);
+
+ /* Init compression */
+ PqStream = zpq_create(res_compressors, n_res_compressors, write_compressed, read_compressed, MyProcPort, NULL, 0);
+ if (!PqStream)
+ {
+ ereport(LOG, (errmsg("failed to initialize the compression stream")));
+ return -1;
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -270,6 +419,9 @@ socket_close(int code, Datum arg)
}
#endif /* ENABLE_GSS */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -941,12 +1093,15 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * nowait parameter toggles non-blocking mode.
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -962,21 +1117,40 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ /*
+ * If streaming compression is enabled then use correspondent
+ * compression read function.
+ */
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
if (r < 0)
{
+ if (r == ZS_DECOMPRESS_ERROR)
+ {
+ char const *msg = zpq_decompress_error(PqStream);
+
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -997,7 +1171,8 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ pgstat_report_network_traffic(r, 0, 0, 0);
+ return r;
}
}
@@ -1012,7 +1187,8 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1031,7 +1207,8 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1048,48 +1225,15 @@ pq_peekbyte(void)
int
pq_getbyte_if_available(unsigned char *c)
{
- int r;
+ int r = 0;
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1110,7 +1254,8 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1144,7 +1289,8 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1168,7 +1314,6 @@ pq_buffer_has_data(void)
return (PqRecvPointer < PqRecvLength);
}
-
/* --------------------------------
* pq_startmsgread - begin reading a message from the client.
*
@@ -1372,13 +1517,24 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered_tx(PqStream) != 0)
+
+ /*
+ * has more data to flush or unsent data in internal compression
+ * buffer
+ */
{
int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
- if (r <= 0)
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1421,12 +1577,12 @@ internal_flush(void)
InterruptPending = 1;
return EOF;
}
+ pgstat_report_network_traffic(0, r, 0, 0);
last_reported_send_errno = 0; /* reset after any successful send */
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
@@ -1443,7 +1599,7 @@ socket_flush_if_writable(void)
int res;
/* Quick exit if nothing to do */
- if (PqSendPointer == PqSendStart)
+ if ((PqSendPointer == PqSendStart) && (zpq_buffered_tx(PqStream) == 0))
return 0;
/* No-op if reentrant call */
@@ -1466,7 +1622,7 @@ socket_flush_if_writable(void)
static bool
socket_is_send_pending(void)
{
- return (PqSendStart < PqSendPointer);
+ return (PqSendStart < PqSendPointer || (zpq_buffered_tx(PqStream) != 0));
}
/* --------------------------------
@@ -1989,3 +2145,16 @@ pq_check_connection(void)
return true;
}
+
+PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+
+Datum
+pg_compression_algorithm(PG_FUNCTION_ARGS)
+{
+ char const *algorithm_name = PqStream ? zpq_compress_algorithm_name(PqStream) : NULL;
+
+ if (algorithm_name)
+ PG_RETURN_TEXT_P(cstring_to_text(algorithm_name));
+ else
+ PG_RETURN_NULL();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 328ecafa8cb..3576869311d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2233,6 +2233,8 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "_pq_.compression") == 0)
+ port->compression_algorithms = pstrdup(valptr);
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4555,6 +4557,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 6ae92784114..dc26b5144bb 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -357,6 +357,9 @@ pgstat_bestart(void)
lbeentry.st_xact_start_timestamp = 0;
lbeentry.st_databaseid = MyDatabaseId;
+ lbeentry.st_tx_raw_bytes = lbeentry.st_rx_raw_bytes =
+ lbeentry.st_tx_compressed_bytes = lbeentry.st_rx_compressed_bytes = 0;
+
/* We have userid for client-backends, wal-sender and bgworker processes */
if (lbeentry.st_backendType == B_BACKEND
|| lbeentry.st_backendType == B_WAL_SENDER
@@ -771,6 +774,33 @@ pgstat_report_xact_timestamp(TimestampTz tstamp)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Report current transaction start timestamp as the specified value.
+ * Zero means there is no active transaction.
+ */
+void
+pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ if (!pgstat_track_activities || !beentry)
+ return;
+
+ /*
+ * Update my status entry, following the protocol of bumping
+ * st_changecount before and after. We use a volatile pointer here to
+ * ensure the compiler doesn't try to get cute.
+ */
+ PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
+
+ beentry->st_rx_raw_bytes += rx_raw_bytes;
+ beentry->st_tx_raw_bytes += tx_raw_bytes;
+ beentry->st_rx_compressed_bytes += rx_compressed_bytes;
+ beentry->st_tx_compressed_bytes += tx_compressed_bytes;
+
+ PGSTAT_END_WRITE_ACTIVITY(beentry);
+}
+
/* ----------
* pgstat_read_current_status() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 5a954c38aca..3acfdc56f5c 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -567,7 +567,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
Datum
pg_stat_get_activity(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_ACTIVITY_COLS 30
+#define PG_STAT_GET_ACTIVITY_COLS 34
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
int pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
@@ -916,6 +916,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[29] = true;
else
values[29] = UInt64GetDatum(beentry->st_query_id);
+ values[30] = beentry->st_rx_raw_bytes;
+ values[31] = beentry->st_tx_raw_bytes;
+ values[32] = beentry->st_rx_compressed_bytes;
+ values[33] = beentry->st_tx_compressed_bytes;
}
else
{
@@ -944,6 +948,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[27] = true;
nulls[28] = true;
nulls[29] = true;
+ nulls[30] = true;
+ nulls[31] = true;
+ nulls[32] = true;
+ nulls[33] = true;
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -1143,6 +1151,46 @@ pg_stat_get_backend_start(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(result);
}
+Datum
+pg_stat_get_network_traffic(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_NETWORK_TRAFFIC_COLS 4
+ TupleDesc tupdesc;
+ Datum values[PG_STAT_NETWORK_TRAFFIC_COLS];
+ bool nulls[PG_STAT_NETWORK_TRAFFIC_COLS];
+ int32 beid = PG_GETARG_INT32(0);
+ PgBackendStatus *beentry;
+
+ if ((beentry = pgstat_fetch_stat_beentry(beid)) == NULL)
+ PG_RETURN_NULL();
+ else if (!HAS_PGSTAT_PERMISSIONS(beentry->st_userid))
+ PG_RETURN_NULL();
+
+ /* Initialise values and NULL flags arrays */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(PG_STAT_NETWORK_TRAFFIC_COLS);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "rx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "tx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "rx_compressed_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "tx_compressed_bytes",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ /* Fill values and NULLs */
+ values[0] = beentry->st_rx_raw_bytes;
+ values[1] = beentry->st_tx_raw_bytes;
+ values[2] = beentry->st_rx_compressed_bytes;
+ values[3] = beentry->st_tx_compressed_bytes;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
Datum
pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 17cfda9d200..f22611700fd 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -51,6 +51,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "common/string.h"
+#include "common/zpq_stream.h"
#include "funcapi.h"
#include "jit/jit.h"
#include "libpq/auth.h"
@@ -197,6 +198,7 @@ static bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
static bool check_log_stats(bool *newval, void **extra, GucSource source);
static bool check_canonical_path(char **newval, void **extra, GucSource source);
static bool check_timezone_abbreviations(char **newval, void **extra, GucSource source);
+static bool check_libpq_compression(char **newval, void **extra, GucSource source);
static void assign_timezone_abbreviations(const char *newval, void *extra);
static void pg_timezone_abbrev_initialize(void);
static const char *show_archive_command(void);
@@ -4589,6 +4591,16 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"libpq_compression", PGC_SIGHUP, CLIENT_CONN_OTHER,
+ gettext_noop("Sets the list of allowed libpq compression algorithms."),
+ NULL
+ },
+ &libpq_compress_algorithms,
+ "on",
+ check_libpq_compression, NULL, NULL
+ },
+
{
{"application_name", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Sets the application name to be reported in statistics and logs."),
@@ -5021,6 +5033,8 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
@@ -11898,6 +11912,22 @@ check_ssl(bool *newval, void **extra, GucSource source)
return true;
}
+static bool
+check_libpq_compression(char **newval, void **extra, GucSource source)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(*newval, &compressors, &n_compressors))
+ {
+ GUC_check_errdetail("Cannot parse the libpq_compression setting.");
+ return false;
+ }
+
+ free(compressors);
+ return true;
+}
+
static bool
check_stage_log_stats(bool *newval, void **extra, GucSource source)
{
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ea9639984cf..6d5dcb2b795 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6661,6 +6661,9 @@ threadRun(void *arg)
int nsocks; /* number of sockets to be waited for */
pg_time_usec_t min_usec;
pg_time_usec_t now = 0; /* set this only if needed */
+ bool buffered_rx = false; /* true if some of the clients has
+ * data left in SSL/ZPQ read
+ * buffers */
/*
* identify which client sockets should be checked for input, and
@@ -6695,6 +6698,9 @@ threadRun(void *arg)
*/
int sock = PQsocket(st->con);
+ /* check if conn has buffered SSL / ZPQ read data */
+ buffered_rx = buffered_rx || PQreadPending(st->con);
+
if (sock < 0)
{
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
@@ -6738,7 +6744,7 @@ threadRun(void *arg)
{
if (nsocks > 0)
{
- rc = wait_on_socket_set(sockets, min_usec);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, min_usec);
}
else /* nothing active, simple sleep */
{
@@ -6747,7 +6753,7 @@ threadRun(void *arg)
}
else /* no explicit delay, wait without timeout */
{
- rc = wait_on_socket_set(sockets, 0);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, 0);
}
if (rc < 0)
@@ -6786,8 +6792,11 @@ threadRun(void *arg)
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
goto done;
}
-
- if (!socket_has_input(sockets, sock, nsocks++))
+ if (PQreadPending(st->con))
+ {
+ nsocks++;
+ }
+ else if (!socket_has_input(sockets, sock, nsocks++))
continue;
}
else if (st->state == CSTATE_FINISHED ||
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index fb3bab94948..c87d86db58e 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -171,6 +171,7 @@ static void print_with_linenumbers(FILE *output, char *lines,
const char *header_keyword);
static void minimal_error_message(PGresult *res);
+static void printCompressionInfo(void);
static void printSSLInfo(void);
static void printGSSInfo(void);
static bool printPsetInfo(const char *param, printQueryOpt *popt);
@@ -633,6 +634,7 @@ exec_command_conninfo(PsqlScanState scan_state, bool active_branch)
printf(_("You are connected to database \"%s\" as user \"%s\" on host \"%s\" at port \"%s\".\n"),
db, PQuser(pset.db), host, PQport(pset.db));
}
+ printCompressionInfo();
printSSLInfo();
printGSSInfo();
}
@@ -3647,6 +3649,27 @@ connection_warnings(bool in_startup)
}
}
+/*
+ * printCompressionInfo
+ *
+ * Print information about used compressor/decompressor
+ */
+static void
+printCompressionInfo(void)
+{
+ char *compressor = PQcompressor(pset.db);
+ char *decompressor = PQdecompressor(pset.db);
+
+ if (compressor != NULL)
+ {
+ printf(_("Compressor %s\n"), compressor);
+ }
+
+ if (decompressor != NULL)
+ {
+ printf(_("Decompressor %s\n"), decompressor);
+ }
+}
/*
* printSSLInfo
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366df..185c96063f6 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,7 +78,9 @@ OBJS_COMMON = \
unicode_norm.o \
username.o \
wait_error.o \
- wchar.o
+ wchar.o \
+ z_stream.o \
+ zpq_stream.o
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
new file mode 100644
index 00000000000..d94914e7dde
--- /dev/null
+++ b/src/common/z_stream.c
@@ -0,0 +1,659 @@
+#include "c.h"
+#include "pg_config.h"
+#include "common/z_stream.h"
+
+/*
+ * Functions implementing streaming compression algorithm
+ */
+typedef struct
+{
+ /*
+ * Name of compression algorithm.
+ */
+ char const *(*name) (void);
+
+ /*
+ * Create new compression stream. level: compression level
+ */
+ void *(*create_compressor) (int level);
+
+ /*
+ * Create new decompression stream.
+ */
+ void *(*create_decompressor) ();
+
+ /*
+ * Decompress up to "src_size" compressed bytes from *src and write up to
+ * "dst_size" raw (decompressed) bytes to *dst. Number of decompressed
+ * bytes written to *dst is stored in *dst_processed. Number of compressed
+ * bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during decompression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * decompressor internal buffers.
+ *
+ * ZS_STREAM_END if encountered end of compressed data stream.
+ *
+ * ZS_DECOMPRESS_ERROR if encountered an error during decompression
+ * attempt.
+ */
+ ssize_t (*decompress) (void *ds, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Compress up to "src_size" raw (non-compressed) bytes from *src and
+ * write up to "dst_size" compressed bytes to *dst. Number of compressed
+ * bytes written to *dst is stored in *dst_processed. Number of
+ * non-compressed bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during compression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * compressor internal buffers.
+ *
+ * ZS_COMPRESS_ERROR if encountered an error during compression attempt.
+ */
+ ssize_t (*compress) (void *cs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Free compression stream created by create_compressor function.
+ */
+ void (*free_compressor) (void *cs);
+
+ /*
+ * Free decompression stream created by create_decompressor function.
+ */
+ void (*free_decompressor) (void *ds);
+
+ /*
+ * Get compressor error message.
+ */
+ char const *(*compress_error) (void *cs);
+
+ /*
+ * Get decompressor error message.
+ */
+ char const *(*decompress_error) (void *ds);
+
+ ssize_t (*end_compression) (void *cs, void *dst, size_t dst_size, size_t *dst_processed);
+} ZAlgorithm;
+
+struct ZStream
+{
+ ZAlgorithm const *algorithm;
+ void *stream;
+ bool not_flushed;
+};
+
+#if HAVE_LIBZ
+
+#include <stdlib.h>
+#include <zlib.h>
+
+
+static void *
+zlib_create_compressor(int level)
+{
+ int rc;
+ z_stream *c_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(c_stream, 0, sizeof(*c_stream));
+ rc = deflateInit(c_stream, level);
+ if (rc != Z_OK)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+zlib_create_decompressor()
+{
+ int rc;
+ z_stream *d_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(d_stream, 0, sizeof(*d_stream));
+ rc = inflateInit(d_stream);
+ if (rc != Z_OK)
+ {
+ free(d_stream);
+ return NULL;
+ }
+ return d_stream;
+}
+
+static ssize_t
+zlib_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *ds = (z_stream *) d_stream;
+ int rc;
+
+ ds->next_in = (Bytef *) src;
+ ds->avail_in = src_size;
+ ds->next_out = (Bytef *) dst;
+ ds->avail_out = dst_size;
+
+ rc = inflate(ds, Z_SYNC_FLUSH);
+ *src_processed = src_size - ds->avail_in;
+ *dst_processed = dst_size - ds->avail_out;
+
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_STREAM_END;
+ }
+ if (rc != Z_OK && rc != Z_BUF_ERROR)
+ {
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+ unsigned deflate_pending = 0;
+
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = (Bytef *) src;
+ cs->avail_in = src_size;
+
+ rc = deflate(cs, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ *dst_processed = dst_size - cs->avail_out;
+ *src_processed = src_size - cs->avail_in;
+
+ deflatePending(cs, &deflate_pending, Z_NULL); /* check if any data left
+ * in deflate buffer */
+ if (deflate_pending > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+zlib_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = NULL;
+ cs->avail_in = 0;
+
+ rc = deflate(cs, Z_STREAM_END);
+ Assert(rc == Z_OK || rc == Z_STREAM_END);
+ *dst_processed = dst_size - cs->avail_out;
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_OK;
+ }
+
+ return ZS_DATA_PENDING;
+}
+
+static void
+zlib_free_compressor(void *c_stream)
+{
+ z_stream *cs = (z_stream *) c_stream;
+
+ if (cs != NULL)
+ {
+ deflateEnd(cs);
+ free(cs);
+ }
+}
+
+static void
+zlib_free_decompressor(void *d_stream)
+{
+ z_stream *ds = (z_stream *) d_stream;
+
+ if (ds != NULL)
+ {
+ inflateEnd(ds);
+ free(ds);
+ }
+}
+
+static char const *
+zlib_error(void *stream)
+{
+ z_stream *zs = (z_stream *) stream;
+
+ return zs->msg;
+}
+
+static char const *
+zlib_name(void)
+{
+ return "zlib";
+}
+
+#endif
+
+#if USE_LZ4
+#include <lz4.h>
+
+#define MESSAGE_MAX_BYTES 819200
+#define RING_BUFFER_BYTES (1024 * 64 + MESSAGE_MAX_BYTES)
+
+typedef struct ZS_LZ4_CStream
+{
+ LZ4_stream_t *stream;
+ int level;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_CStream;
+
+typedef struct ZS_LZ4_DStream
+{
+ LZ4_streamDecode_t *stream;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_DStream;
+
+static void *
+lz4_create_compressor(int level)
+{
+ ZS_LZ4_CStream *c_stream = (ZS_LZ4_CStream *) malloc(sizeof(ZS_LZ4_CStream));
+ if (c_stream == NULL) {
+ return NULL;
+ }
+ c_stream->stream = LZ4_createStream();
+ c_stream->level = level;
+ c_stream->buf_pos = 0;
+ if (c_stream->stream == NULL)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+lz4_create_decompressor()
+{
+ ZS_LZ4_DStream *d_stream = (ZS_LZ4_DStream *) malloc(sizeof(ZS_LZ4_DStream));
+ if (d_stream == NULL) {
+ return NULL;
+ }
+
+ d_stream->stream = LZ4_createStreamDecode();
+ d_stream->buf_pos = 0;
+ if (d_stream->stream == NULL)
+ {
+ free(d_stream);
+ return NULL;
+ }
+
+ return d_stream;
+}
+
+static ssize_t
+lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+ Assert(src_size > 0);
+
+ char* const decPtr = &ds->buf[ds->buf_pos];
+
+ const int decBytes = LZ4_decompress_safe_continue(
+ ds->stream, src, decPtr, (int) src_size, (int) dst_size);
+
+ Assert(decBytes > 0);
+
+ *dst_processed = decBytes;
+ *src_processed = src_size;
+
+ memcpy(dst, decPtr, decBytes); /* write msg length */
+
+ ds->buf_pos += decBytes;
+ if (ds->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ ds->buf_pos = 0;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ src_size = Min(MESSAGE_MAX_BYTES, src_size);
+
+ Assert(src_size > 0);
+
+ memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
+
+ Assert(dst_size >= LZ4_compressBound(src_size));
+
+ const int cmpBytes = LZ4_compress_fast_continue(
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, 1);
+
+ Assert(cmpBytes > 0);
+ Assert(cmpBytes <= MESSAGE_MAX_BYTES);
+
+ *dst_processed = cmpBytes;
+ *src_processed = src_size;
+
+ cs->buf_pos += src_size;
+ if (cs->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ cs->buf_pos = 0;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+lz4_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ *dst_processed = 0;
+ return ZS_OK;
+}
+
+static void
+lz4_free_compressor(void *c_stream)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ if (cs->stream != NULL) {
+ LZ4_freeStream(cs->stream);
+ }
+ free(cs);
+ }
+}
+
+static void
+lz4_free_decompressor(void *d_stream)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ if (ds->stream != NULL) {
+ LZ4_freeStreamDecode(ds->stream);
+ }
+ free(ds);
+ }
+}
+
+static char const *
+lz4_error(void *stream)
+{
+ // lz4 doesn't have any explicit API to get the error names
+ return "NO_MSG";
+}
+
+static char const *
+lz4_name(void)
+{
+ return "lz4";
+}
+
+#endif
+
+static char const *
+no_compression_name(void)
+{
+ return NULL;
+}
+
+/*
+ * Array with all supported compression algorithms.
+ */
+static ZAlgorithm const zs_algorithms[] =
+{
+#if HAVE_LIBZ
+ {zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
+#endif
+#if USE_LZ4
+ {lz4_name, lz4_create_compressor, lz4_create_decompressor, lz4_decompress, lz4_compress, lz4_free_compressor, lz4_free_decompressor, lz4_error, lz4_error, lz4_end},
+#endif
+ {no_compression_name}
+};
+
+inline bool
+zs_is_valid_impl_id(unsigned int id)
+{
+ return id >= 0 && id < (sizeof(zs_algorithms) / sizeof(*zs_algorithms));
+}
+
+static ssize_t
+zs_init_compressor(ZStream * zs, unsigned int c_alg_impl, int c_level)
+{
+ if (!zs_is_valid_impl_id(c_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[c_alg_impl];
+ zs->stream = zs->algorithm->create_compressor(c_level);
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+static ssize_t
+zs_init_decompressor(ZStream * zs, unsigned int d_alg_impl)
+{
+ if (!zs_is_valid_impl_id(d_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[d_alg_impl];
+ zs->stream = zs->algorithm->create_decompressor();
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Index of used compression algorithm in zs_algorithms array.
+ */
+ZStream *
+zs_create_compressor(unsigned int c_alg_impl, int c_level)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_compressor(zs, c_alg_impl, c_level))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ZStream *
+zs_create_decompressor(unsigned int d_alg_impl)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_decompressor(zs, d_alg_impl))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ssize_t
+zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *src_processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->decompress(zs->stream,
+ src, src_size, src_processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+
+ if (rc == ZS_OK || rc == ZS_INCOMPLETE_SRC || rc == ZS_STREAM_END) {
+ return rc;
+ }
+
+ return ZS_DECOMPRESS_ERROR;
+}
+
+ssize_t
+zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->compress(zs->stream,
+ buf, size, processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+void
+zs_compressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_compressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+void
+zs_decompressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_decompressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+ssize_t
+zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *dst_processed = 0;
+
+ rc = zs->algorithm->end_compression(zs->stream, dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+char const *
+zs_compress_error(ZStream * zs)
+{
+ return zs->algorithm->compress_error(zs->stream);
+}
+
+char const *
+zs_decompress_error(ZStream * zs)
+{
+ return zs->algorithm->decompress_error(zs->stream);
+}
+
+bool
+zs_buffered(ZStream * zs)
+{
+ return zs ? zs->not_flushed : 0;
+}
+
+
+/*
+ * Get list of the supported algorithms.
+ */
+char **
+zs_get_supported_algorithms(void)
+{
+ size_t n_algorithms = sizeof(zs_algorithms) / sizeof(*zs_algorithms);
+ char **algorithm_names = malloc(n_algorithms * sizeof(char *));
+
+ for (size_t i = 0; i < n_algorithms; i++)
+ {
+ algorithm_names[i] = (char *) zs_algorithms[i].name();
+ }
+
+ return algorithm_names;
+}
+
+char const *
+zs_compress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
+
+char const *
+zs_decompress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 00000000000..294ad249ce7
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,1001 @@
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+#include <unistd.h>
+#include <math.h>
+
+#include "common/zpq_stream.h"
+#include "pg_config.h"
+#include "port/pg_bswap.h"
+
+/* log warnings on backend */
+#ifndef FRONTEND
+#define pg_log_warning(...) elog(WARNING, __VA_ARGS__)
+#else
+#define pg_log_warning(...) (void)0
+#endif
+
+/* ZpqBuffer size, in bytes */
+#define ZPQ_BUFFER_SIZE 8192000
+/* CompressedData msg_type */
+#define ZPQ_COMPRESSED_MSG_TYPE 'm'
+/* SetCompressionMethod msg_type */
+#define ZPQ_SET_COMPRESSION_MSG_TYPE 'k'
+
+#define ZPQ_COMPRESS_THRESHOLD 60
+
+typedef struct ZpqBuffer ZpqBuffer;
+
+
+/* ZpqBuffer used as RX/TX buffer in ZpqStream */
+struct ZpqBuffer
+{
+ char buf[ZPQ_BUFFER_SIZE];
+ size_t size; /* current size of buf */
+ size_t pos; /* current position in buf, in range [0, size] */
+};
+
+static inline void
+zpq_buf_init(ZpqBuffer * zb)
+{
+ zb->size = 0;
+ zb->pos = 0;
+}
+
+static inline size_t
+zpq_buf_left(ZpqBuffer * zb)
+{
+ Assert(zb->buf);
+ return ZPQ_BUFFER_SIZE - zb->size;
+}
+
+static inline size_t
+zpq_buf_unread(ZpqBuffer * zb)
+{
+ return zb->size - zb->pos;
+}
+
+static inline char *
+zpq_buf_size(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->size;
+}
+
+static inline char *
+zpq_buf_pos(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->pos;
+}
+
+static inline void
+zpq_buf_size_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->size += value;
+}
+
+static inline void
+zpq_buf_pos_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->pos += value;
+}
+
+static inline void
+zpq_buf_reuse(ZpqBuffer * zb)
+{
+ size_t unread = zpq_buf_unread(zb);
+
+ if (unread > 5) /* can read message header, don't do anything */
+ return;
+ if (unread == 0)
+ {
+ zb->size = 0;
+ zb->pos = 0;
+ return;
+ }
+ memmove(zb->buf, zb->buf + zb->pos, unread);
+ zb->size = unread;
+ zb->pos = 0;
+}
+
+struct ZpqStream
+{
+ ZStream *c_stream; /* underlying compression stream */
+ ZStream *d_stream; /* underlying decompression stream */
+
+ size_t tx_total; /* amount of bytes sent to tx_func */
+
+ size_t tx_total_raw; /* amount of bytes received by zpq_write */
+ size_t rx_total; /* amount of bytes read by rx_func */
+ size_t rx_total_raw; /* amount of bytes returned by zpq_write */
+ bool is_compressing; /* current compression state */
+
+ bool is_decompressing; /* current decompression state */
+ size_t rx_msg_bytes_left; /* number of bytes left to process without
+ * changing the decompression state */
+ size_t tx_msg_bytes_left; /* number of bytes left to process without
+ * changing the compression state */
+
+ ZpqBuffer rx_in; /* buffer for unprocessed data read by rx_func */
+ ZpqBuffer tx_in; /* buffer for unprocessed data consumed by
+ * zpq_write */
+ ZpqBuffer tx_out; /* buffer for processed data waiting for send
+ * via tx_func */
+
+ zpq_rx_func rx_func;
+ zpq_tx_func tx_func;
+ void *arg;
+
+ zpq_compressor *compressors; /* compressors array holds the available
+ * compressors to use for
+ * compression/decompression */
+ size_t n_compressors; /* size of the compressors array */
+ int compress_alg_idx; /* index of the active compression
+ * algorithm */
+ int decompress_alg_idx; /* index of the active decompression
+ * algorithm */
+ int compressor_by_msg_type[256]; /* map to choose a compressor
+ * by the protocol message
+ * type */
+
+ bool reading_set_compression; /* utility marker indicating
+ * partial SetCompressionMethod
+ * read */
+};
+
+/*
+ * Message compression map defines the logic for choosing the compressor
+ * based on the protocol message type. Currently, it is a basic prototype to demonstrate
+ * the capabilities of the on-the-fly compression switch.
+ */
+static inline void
+zpq_build_msg_compression_map(ZpqStream * zpq)
+{
+ int i;
+
+ for (i = 0; i < 256; i++)
+ {
+ zpq->compressor_by_msg_type[i] = -1;
+ }
+
+ for (i = 0; i < zpq->n_compressors; i++)
+ {
+ /* compress CopyData, DataRow and Query messages */
+ if (zpq->compressor_by_msg_type['d'] == -1) {
+ zpq->compressor_by_msg_type['d'] = i;
+ }
+ if (zpq->compressor_by_msg_type['D'] == -1) {
+ zpq->compressor_by_msg_type['D'] = i;
+ }
+ if (zpq->compressor_by_msg_type['Q'] == -1) {
+ zpq->compressor_by_msg_type['Q'] = i;
+ }
+ }
+}
+
+/*
+ * Choose the index of compressor to use for the message of msg_type with msg_len.
+ * Return values:
+ * - the non-negative index of zpq->compressors array
+ * - -1, if message should not be compressed
+ */
+static inline int
+zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
+ {
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
+ }
+ return -1;
+}
+
+/*
+ * Check if should compress message of msg_type with msg_len.
+ * Return true if should, false if should not.
+ */
+static inline bool
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
+}
+
+/*
+ * Check if message is a CompressedData.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_compressed_msg(char msg_type)
+{
+ return msg_type == ZPQ_COMPRESSED_MSG_TYPE;
+}
+
+/*
+ * Check if message is a SetCompressionMethod.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_set_compression_msg(char msg_type)
+{
+ return msg_type == ZPQ_SET_COMPRESSION_MSG_TYPE;
+}
+
+ZpqStream *
+zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size)
+{
+ ZpqStream *zpq;
+
+ /* zpqStream needs at least one compressor */
+ if (n_compressors == 0 || compressors == NULL)
+ {
+ return NULL;
+ }
+ zpq = (ZpqStream *) malloc(sizeof(ZpqStream));
+
+ zpq->compressors = compressors;
+ zpq->n_compressors = n_compressors;
+ zpq->compress_alg_idx = -1;
+ zpq->decompress_alg_idx = -1;
+
+ zpq->is_compressing = false;
+ zpq->is_decompressing = false;
+ zpq->rx_msg_bytes_left = 0;
+ zpq->tx_msg_bytes_left = 0;
+ zpq_buf_init(&zpq->tx_in);
+
+ zpq->tx_total = 0;
+ zpq->tx_total_raw = 0;
+ zpq->rx_total = 0;
+ zpq->rx_total_raw = 0;
+
+ zpq_buf_init(&zpq->rx_in);
+ zpq_buf_size_advance(&zpq->rx_in, rx_data_size);
+ Assert(rx_data_size < ZPQ_BUFFER_SIZE);
+ memcpy(zpq->rx_in.buf, rx_data, rx_data_size);
+
+ zpq_buf_init(&zpq->tx_out);
+
+ zpq->rx_func = rx_func;
+ zpq->tx_func = tx_func;
+ zpq->arg = arg;
+ zpq->reading_set_compression = false;
+ zpq->c_stream = NULL;
+ zpq->d_stream = NULL;
+
+ zpq_build_msg_compression_map(zpq);
+
+ return zpq;
+}
+
+/* Compress up to src_size bytes from *src into CompressedData and write it to the tx buffer.
+ * Returns ZS_OK on success, ZS_COMPRESS_ERROR if encountered a compression error. */
+static inline ssize_t
+zpq_write_compressed_message(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ size_t compressed_len;
+ ssize_t rc;
+ uint32 size;
+
+ /* check if have enough space */
+ if (zpq_buf_left(&zpq->tx_out) <= 5)
+ {
+ /* too little space for CompressedData, abort */
+ *src_processed = 0;
+ return ZS_OK;
+ }
+
+ compressed_len = 0;
+ rc = zs_write(zpq->c_stream, src, src_size, src_processed,
+ zpq_buf_size(&zpq->tx_out) + 5, zpq_buf_left(&zpq->tx_out) - 5, &compressed_len);
+
+ if (compressed_len > 0)
+ {
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_COMPRESSED_MSG_TYPE;
+ size = pg_hton32(compressed_len + 4);
+
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &size, sizeof(uint32)); /* write msg length */
+ compressed_len += 5; /* append header length to compressed data
+ * length */
+ }
+
+ zpq->tx_total_raw += *src_processed;
+ zpq->tx_total += compressed_len;
+ zpq_buf_size_advance(&zpq->tx_out, compressed_len);
+ return rc;
+}
+
+/* Copy the data directly from *src to the tx buffer */
+static void
+zpq_write_uncompressed(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ src_size = Min(zpq_buf_left(&zpq->tx_out), src_size);
+ memcpy(zpq_buf_size(&zpq->tx_out), src, src_size);
+
+ zpq->tx_total_raw += src_size;
+ zpq->tx_total += src_size;
+ zpq_buf_size_advance(&zpq->tx_out, src_size);
+ *src_processed = src_size;
+}
+
+static ssize_t
+zpq_write_set_compression_msg(ZpqStream * zpq, int new_compress_idx)
+{
+ uint32 len;
+ uint8 idx;
+
+ /*
+ * check if have enough space: msg_type(1 byte) + msg_len(4 bytes) +
+ * compress_alg_idx(1 byte)
+ */
+ if (zpq_buf_left(&zpq->tx_out) < 6)
+ {
+ return -1;
+ }
+
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_SET_COMPRESSION_MSG_TYPE;
+ len = pg_hton32(5);
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &len, sizeof(uint32)); /* write msg length */
+
+ /* currently we expect idx to be in range [0, 255] */
+ Assert(new_compress_idx >= 0 && new_compress_idx <= UINT8_MAX);
+ idx = (uint8) new_compress_idx;
+ memcpy(zpq_buf_size(&zpq->tx_out) + 5, &idx, sizeof(uint8)); /* write
+ * new_compress_idx */
+
+ zpq->tx_total_raw += 6;
+ zpq->tx_total += 6;
+ zpq_buf_size_advance(&zpq->tx_out, 6);
+ return 0;
+}
+
+/* Determine if should compress the next message and change the current compression state */
+static ssize_t
+zpq_toggle_compression(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ int new_compress_idx = zpq_choose_compressor(zpq, msg_type, msg_len);
+ bool should_compress = new_compress_idx != -1;
+
+ /*
+ * negative new_compress_idx indicates that we should not compress this
+ * message
+ */
+ if (should_compress)
+ {
+ /*
+ * if the new compressor does not match the current one, process the
+ * switch
+ */
+ if (zpq->compress_alg_idx != new_compress_idx)
+ {
+ if (zpq_write_set_compression_msg(zpq, new_compress_idx))
+ {
+ /*
+ * come back later when we can write the entire
+ * SetCompressionMethod message
+ */
+ return 0;
+ }
+
+ zs_compressor_free(zpq->c_stream);
+ zpq->c_stream = zs_create_compressor(zpq->compressors[new_compress_idx].impl, zpq->compressors[new_compress_idx].level);
+ if (zpq->c_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->compress_alg_idx = new_compress_idx;
+ }
+ }
+
+ zpq->is_compressing = should_compress;
+ zpq->tx_msg_bytes_left = msg_len + 1;
+ return 0;
+}
+
+/*
+ * Internal write function. Reads the data from *src buffer,
+ * determines the postgres messages type and length.
+ * If message matches the compression criteria, it wraps the message into
+ * CompressedData. Otherwise, leaves the message unchanged.
+ * If *src data ends with incomplete message header, this function is not
+ * going to read this message header.
+ * Returns number of written raw bytes or error code.
+ * In the last case number of bytes written is stored in *processed.
+ */
+static ssize_t
+zpq_write_internal(ZpqStream * zpq, void const *src, size_t src_size, size_t *processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ do
+ {
+ /*
+ * try to read ahead the next message types and increase
+ * tx_msg_bytes_left, if possible
+ */
+ while (zpq->tx_msg_bytes_left > 0 && src_size - src_pos >= zpq->tx_msg_bytes_left + 5)
+ {
+ char msg_type = *((char *) src + src_pos + zpq->tx_msg_bytes_left);
+ uint32 msg_len;
+
+ memcpy(&msg_len, (char *) src + src_pos + zpq->tx_msg_bytes_left + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ if (zpq_should_compress(zpq, msg_type, msg_len) != zpq->is_compressing)
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+ zpq->tx_msg_bytes_left += msg_len + 1;
+ }
+
+ /*
+ * Write CompressedData if currently is compressing or have some
+ * buffered data left in underlying compression stream
+ */
+ if (zs_buffered(zpq->c_stream) || (zpq->is_compressing && zpq->tx_msg_bytes_left > 0))
+ {
+ size_t buf_processed = 0;
+ size_t to_compress = Min(zpq->tx_msg_bytes_left, src_size - src_pos);
+
+ rc = zpq_write_compressed_message(zpq, (char *) src + src_pos, to_compress, &buf_processed);
+ src_pos += buf_processed;
+ zpq->tx_msg_bytes_left -= buf_processed;
+
+ if (rc != ZS_OK)
+ {
+ *processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * If not going to compress the data from *src, just write it
+ * uncompressed.
+ */
+ else if (zpq->tx_msg_bytes_left > 0)
+ { /* determine next message type */
+ size_t copy_len = Min(src_size - src_pos, zpq->tx_msg_bytes_left);
+ size_t copy_processed = 0;
+
+ zpq_write_uncompressed(zpq, (char *) src + src_pos, copy_len, ©_processed);
+ src_pos += copy_processed;
+ zpq->tx_msg_bytes_left -= copy_processed;
+ }
+
+ /*
+ * Reached the compression toggle point, fetch next message header to
+ * determine compression state.
+ */
+ else
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ if (src_size - src_pos < 5)
+ {
+ /*
+ * must return here because we can't continue without full
+ * message header
+ */
+ *processed = src_pos;
+ return ZPQ_INCOMPLETE_HEADER;
+ }
+
+ msg_type = *((char *) src + src_pos);
+ memcpy(&msg_len, (char *) src + src_pos + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ rc = zpq_toggle_compression(zpq, msg_type, msg_len);
+ if (rc)
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * repeat sending while there is some data in input or internal
+ * compression buffer
+ */
+ } while (src_pos < src_size && zpq_buf_left(&zpq->tx_out) > 6);
+
+ return src_pos;
+}
+
+ssize_t
+zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ /* try to process as much data as possible before calling the tx_func */
+ while (zpq_buf_left(&zpq->tx_out) > 6)
+ {
+ size_t copy_len = Min(zpq_buf_left(&zpq->tx_in), src_size - src_pos);
+ size_t processed;
+
+ memcpy(zpq_buf_size(&zpq->tx_in), (char *) src + src_pos, copy_len);
+ zpq_buf_size_advance(&zpq->tx_in, copy_len);
+ src_pos += copy_len;
+
+ if (zpq_buf_unread(&zpq->tx_in) == 0 && !zs_buffered(zpq->c_stream))
+ {
+ break;
+ }
+
+ processed = 0;
+
+ rc = zpq_write_internal(zpq, zpq_buf_pos(&zpq->tx_in), zpq_buf_unread(&zpq->tx_in), &processed);
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, rc);
+ zpq_buf_reuse(&zpq->tx_in);
+ }
+ else
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, processed);
+ zpq_buf_reuse(&zpq->tx_in);
+ if (rc == ZPQ_INCOMPLETE_HEADER)
+ {
+ break;
+ }
+ *src_processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * call the tx_func if have any bytes to send
+ */
+ while (zpq_buf_unread(&zpq->tx_out))
+ {
+ rc = zpq->tx_func(zpq->arg, zpq_buf_pos(&zpq->tx_out), zpq_buf_unread(&zpq->tx_out));
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_out, rc);
+ }
+ else
+ {
+ *src_processed = src_pos;
+ zpq_buf_reuse(&zpq->tx_out);
+ return rc;
+ }
+ }
+
+ zpq_buf_reuse(&zpq->tx_out);
+ return src_pos;
+}
+
+/* Decompress bytes from RX buffer and write up to dst_len of uncompressed data to *dst.
+ * Returns:
+ * ZS_OK on success,
+ * ZS_STREAM_END if reached end of compressed chunk
+ * ZS_DECOMPRESS_ERROR if encountered a decompression error */
+static inline ssize_t
+zpq_read_compressed_message(ZpqStream * zpq, char *dst, size_t dst_len, size_t *dst_processed)
+{
+ size_t rx_processed = 0;
+ ssize_t rc;
+ size_t read_len = Min(zpq->rx_msg_bytes_left, zpq_buf_unread(&zpq->rx_in));
+
+ Assert(read_len == zpq->rx_msg_bytes_left);
+ rc = zs_read(zpq->d_stream, zpq_buf_pos(&zpq->rx_in), read_len, &rx_processed,
+ dst, dst_len, dst_processed);
+
+ zpq_buf_pos_advance(&zpq->rx_in, rx_processed);
+ zpq->rx_total_raw += *dst_processed;
+ zpq->rx_msg_bytes_left -= rx_processed;
+ return rc;
+}
+
+/* Copy up to dst_len bytes from rx buffer to *dst.
+ * Returns amount of bytes copied. */
+static inline size_t
+zpq_read_uncompressed(ZpqStream * zpq, char *dst, size_t dst_len)
+{
+ size_t copy_len;
+
+ Assert(zpq_buf_unread(&zpq->rx_in) > 0);
+ copy_len = Min(zpq->rx_msg_bytes_left, Min(zpq_buf_unread(&zpq->rx_in), dst_len));
+
+ memcpy(dst, zpq_buf_pos(&zpq->rx_in), copy_len);
+
+ zpq_buf_pos_advance(&zpq->rx_in, copy_len);
+ zpq->rx_total_raw += copy_len;
+ zpq->rx_msg_bytes_left -= copy_len;
+ return copy_len;
+}
+
+/* Determine if should decompress the next message and
+ * change the current decompression state */
+static inline void
+zpq_toggle_decompression(ZpqStream * zpq)
+{
+ uint32 msg_len;
+ char msg_type = *zpq_buf_pos(&zpq->rx_in);
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+
+ if (zpq_is_set_compression_msg(msg_type))
+ {
+ Assert(msg_len == 5);
+ zpq->reading_set_compression = true;
+ /* set compression message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ }
+ else
+ {
+ zpq->is_decompressing = zpq_is_compressed_msg(msg_type);
+ zpq->rx_msg_bytes_left = msg_len + 1;
+
+ if (zpq->is_decompressing)
+ {
+ /* compressed message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ zpq->rx_msg_bytes_left -= 5;
+ }
+ }
+}
+
+static inline ssize_t
+zpq_process_switch(ZpqStream * zpq)
+{
+ uint8 algorithm_idx;
+
+ if (zpq_buf_unread(&zpq->rx_in) < 1)
+ {
+ return 0;
+ }
+
+ algorithm_idx = *zpq_buf_pos(&zpq->rx_in);
+
+ zpq_buf_pos_advance(&zpq->rx_in, 1);
+ zpq->reading_set_compression = false;
+
+ if (algorithm_idx != zpq->decompress_alg_idx)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ zpq->d_stream = zs_create_decompressor(zpq->compressors[algorithm_idx].impl);
+ if (zpq->d_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->decompress_alg_idx = algorithm_idx;
+ }
+
+ return 0;
+}
+
+ssize_t
+zpq_read(ZpqStream * zpq, void *dst, size_t dst_size)
+{
+ size_t dst_pos = 0;
+ size_t dst_processed = 0;
+ ssize_t rc;
+
+ /* Read until some data fetched */
+ while (dst_pos == 0)
+ {
+ zpq_buf_reuse(&zpq->rx_in);
+
+ if (!zpq_buffered_rx(zpq) || (zpq->is_decompressing && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left))
+ {
+ rc = zpq->rx_func(zpq->arg, zpq_buf_size(&zpq->rx_in), zpq_buf_left(&zpq->rx_in));
+ if (rc > 0) /* read fetches some data */
+ {
+ zpq->rx_total += rc;
+ zpq_buf_size_advance(&zpq->rx_in, rc);
+ }
+ else /* read failed */
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * try to read ahead the next message types and increase
+ * rx_msg_bytes_left, if possible (ONLY UNCOMPRESSED MESSAGES)
+ */
+ while (!zpq->is_decompressing && zpq->rx_msg_bytes_left > 0 && (zpq_buf_unread(&zpq->rx_in) >= zpq->rx_msg_bytes_left + 5))
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ msg_type = *(zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left);
+ if (zpq_is_compressed_msg(msg_type) || zpq_is_set_compression_msg(msg_type))
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left + 1, 4);
+ zpq->rx_msg_bytes_left += pg_ntoh32(msg_len) + 1;
+ }
+
+
+ if (zpq->rx_msg_bytes_left > 0 || zs_buffered(zpq->d_stream))
+ {
+ dst_processed = 0;
+ if (zpq->is_decompressing || zs_buffered(zpq->d_stream))
+ {
+ if (!zs_buffered(zpq->d_stream) && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left) {
+ /* prefer to read only the fully compressed messages or read if some data is buffered */
+ continue;
+ }
+ rc = zpq_read_compressed_message(zpq, dst, dst_size - dst_pos, &dst_processed);
+ dst_pos += dst_processed;
+ if (rc == ZS_STREAM_END)
+ {
+ continue;
+ }
+ if (rc != ZS_OK)
+ {
+ return rc;
+ }
+ }
+ else
+ dst_pos += zpq_read_uncompressed(zpq, dst, dst_size - dst_pos);
+ }
+ else if (zpq->reading_set_compression)
+ {
+ zpq_process_switch(zpq);
+ }
+ else if (zpq_buf_unread(&zpq->rx_in) >= 5)
+ zpq_toggle_decompression(zpq);
+ }
+ return dst_pos;
+}
+
+bool
+zpq_buffered_rx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->rx_in) >= 5 || (zpq_buf_unread(&zpq->rx_in) > 0 && zpq->rx_msg_bytes_left > 0) ||
+ zs_buffered(zpq->d_stream) : 0;
+}
+
+bool
+zpq_buffered_tx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->tx_in) >= 5 || (zpq_buf_unread(&zpq->tx_in) > 0 && zpq->tx_msg_bytes_left > 0) || zpq_buf_unread(&zpq->tx_out) > 0 ||
+ zs_buffered(zpq->c_stream) : 0;
+}
+
+void
+zpq_free(ZpqStream * zpq)
+{
+ if (zpq)
+ {
+ if (zpq->c_stream)
+ {
+ zs_compressor_free(zpq->c_stream);
+ }
+ if (zpq->d_stream)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ }
+ free(zpq);
+ }
+}
+
+char const *
+zpq_compress_error(ZpqStream * zpq)
+{
+ return zs_compress_error(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_error(ZpqStream * zpq)
+{
+ return zs_decompress_error(zpq->d_stream);
+}
+
+char const *
+zpq_compress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_compress_algorithm_name(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_decompress_algorithm_name(zpq->d_stream);
+}
+
+bool
+zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int i;
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *protocol_extension = strchr(val, ';');
+
+ *compressors = NULL;
+ *n_compressors = 0;
+
+ /* No protocol extensions are currently supported */
+ if (protocol_extension)
+ *protocol_extension = '\0';
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ if (pg_strcasecmp(val, "true") == 0 ||
+ pg_strcasecmp(val, "yes") == 0 ||
+ pg_strcasecmp(val, "on") == 0 ||
+ pg_strcasecmp(val, "any") == 0 ||
+ pg_strcasecmp(val, "1") == 0)
+ {
+ /* return all available compressors */
+ *n_compressors = n_supported_algorithms;
+
+ if (n_supported_algorithms)
+ {
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+ for (i = 0; i < n_supported_algorithms; i++)
+ {
+ (*compressors)[i].impl = i;
+ (*compressors)[i].level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ }
+ }
+ return true;
+ }
+
+ if (*val == 0 ||
+ pg_strcasecmp(val, "false") == 0 ||
+ pg_strcasecmp(val, "no") == 0 ||
+ pg_strcasecmp(val, "off") == 0 ||
+ pg_strcasecmp(val, "0") == 0)
+ {
+ /* Compression is disabled */
+ return true;
+ }
+
+
+ return zpq_deserialize_compressors(val, compressors, n_compressors);
+}
+
+bool
+zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int selected_alg_mask = 0; /* bitmask of already selected
+ * algorithms to avoid duplicates in
+ * compressors */
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *c_string_dup = strdup(c_string); /* following parsing can
+ * modify the string */
+ char *p = c_string_dup;
+
+ *n_compressors = 0;
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+
+ while (*p != '\0')
+ {
+ char *sep = strchr(p, ',');
+ char *col;
+ int compression_level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ bool found;
+
+ if (sep != NULL)
+ *sep = '\0';
+
+ col = strchr(p, ':');
+ if (col != NULL)
+ {
+ *col = '\0';
+ if (sscanf(col + 1, "%d", &compression_level) != 1)
+ {
+ pg_log_warning("invalid compression level %s in compression option '%s'", col + 1, p);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+ }
+ found = false;
+ for (int i = 0; supported_algorithms[i] != NULL; i++)
+ {
+ if (pg_strcasecmp(p, supported_algorithms[i]) == 0)
+ {
+ if (selected_alg_mask & (1 << i))
+ {
+ /* duplicates are not allowed */
+ pg_log_warning("duplicate algorithm %s in compressors string %s", p, c_string);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+
+ (*compressors)[*n_compressors].impl = i;
+ (*compressors)[*n_compressors].level = compression_level;
+
+ selected_alg_mask |= 1 << i;
+ *n_compressors += 1;
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ pg_log_warning("algorithm %s is not supported", p);
+ }
+ if (sep)
+ p = sep + 1;
+ else
+ break;
+ }
+
+ if (*n_compressors == 0)
+ {
+ free(*compressors);
+ *compressors = NULL;
+ }
+ free(c_string_dup);
+ return true;
+}
+
+char *
+zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors)
+{
+ char *res;
+ char *p;
+ size_t i;
+ size_t total_len = 0;
+ char **supported_algorithms = zs_get_supported_algorithms();
+
+ if (n_compressors == 0)
+ {
+ return NULL;
+ }
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ size_t level_len;
+
+ if (!zs_is_valid_impl_id(compressors[i].impl))
+ {
+ pg_log_warning("algorithm impl_id %d is incorrect", compressors[i].impl);
+ return NULL;
+ }
+
+ /* determine the length of the compression level string */
+ level_len = compressors[i].level == 0 ? 1 : (int) floor(log10(abs(compressors[i].level))) + 1;
+ if (compressors[i].level < 0)
+ {
+ level_len += 1; /* add the leading "-" */
+ }
+
+ /*
+ * single entry looks like "alg_name:compression_level," so +2 is for
+ * ":" and "," symbols
+ */
+ total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
+ }
+
+ res = p = malloc(total_len);
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ p += sprintf(p, "%s:%d,", supported_algorithms[compressors[i].impl], compressors[i].level);
+ }
+ p[-1] = '\0';
+ return res;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index eac2a277490..53aa3f52743 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5334,9 +5334,9 @@
proname => 'pg_stat_get_activity', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'int4',
- proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8}',
- proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id}',
+ proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
prosrc => 'pg_stat_get_activity' },
{ oid => '3318',
descr => 'statistics: information about progress of backends running maintenance command',
@@ -5631,6 +5631,14 @@
proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,wal_write,wal_sync,wal_write_time,wal_sync_time,stats_reset}',
prosrc => 'pg_stat_get_wal' },
+{ oid => '9598', descr => 'statistics: information about network traffic',
+ proname => 'pg_stat_get_network_traffic', proisstrict => 'f', provolatile => 's',
+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o}',
+ proargnames => '{_beid,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
+ prosrc => 'pg_stat_get_network_traffic' },
+
{ oid => '2306', descr => 'statistics: information about SLRU caches',
proname => 'pg_stat_get_slru', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
@@ -5772,6 +5780,10 @@
proname => 'pg_tablespace_location', provolatile => 's', prorettype => 'text',
proargtypes => 'oid', prosrc => 'pg_tablespace_location' },
+{ oid => '9257', descr => 'connection compression algorithm',
+ proname => 'pg_compression_algorithm', provolatile => 's', prorettype => 'text',
+ proargtypes => '', prosrc => 'pg_compression_algorithm' },
+
{ oid => '1946',
descr => 'convert bytea value into some ascii-only text string',
proname => 'encode', prorettype => 'text', proargtypes => 'bytea text',
diff --git a/src/include/common/z_stream.h b/src/include/common/z_stream.h
new file mode 100644
index 00000000000..8229035978f
--- /dev/null
+++ b/src/include/common/z_stream.h
@@ -0,0 +1,107 @@
+/*
+ * z_stream.h
+ * Streaming compression
+ */
+
+
+#ifndef Z_STREAM_H
+#define Z_STREAM_H
+
+#include <stdlib.h>
+
+#define ZS_OK (0)
+#define ZS_IO_ERROR (-1)
+#define ZS_DECOMPRESS_ERROR (-2)
+#define ZS_COMPRESS_ERROR (-3)
+#define ZS_STREAM_END (-4)
+#define ZS_DATA_PENDING (-5)
+#define ZS_INCOMPLETE_SRC (-6) /* cannot decompress unless full src message is fetched */
+#define ZS_EXPAND_DST (-7) /* cannot compress: dst buffer is too small for given src size */
+
+struct ZStream;
+typedef struct ZStream ZStream;
+
+#endif
+
+/*
+ * Create compression stream for sending compressed data.
+ * c_alg_impl: index of chosen compression algorithm
+ * c_level: compression c_level
+ */
+extern ZStream * zs_create_compressor(unsigned int c_alg_impl, int c_level);
+
+/*
+ * Create decompression stream for reading compressed data.
+ * d_alg_impl: index of chosen decompression algorithm
+ */
+extern ZStream * zs_create_decompressor(unsigned int d_alg_impl);
+
+/*
+ * Read up to "size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZS_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Write up to "size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZS_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *processed.
+ */
+extern ssize_t zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zs_decompress_error(ZStream * zs);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zs_compress_error(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal rx decompression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal tx compression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * End the compression stream.
+ */
+extern ssize_t zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Free stream created by zs_create_compressor function.
+ */
+extern void zs_compressor_free(ZStream * zs);
+
+/*
+ * Free stream created by zs_create_decompressor function.
+ */
+extern void zs_decompressor_free(ZStream * zs);
+
+/*
+ * Get the name of chosen compression algorithm.
+ */
+extern char const *zs_compress_algorithm_name(ZStream * zs);
+
+/*
+ * Get the name of chosen decompression algorithm.
+ */
+extern char const *zs_decompress_algorithm_name(ZStream * zs);
+
+/*
+ Returns zero terminated array with compression algorithms names
+*/
+extern char **zs_get_supported_algorithms(void);
+
+/*
+ Returns true if provided id is a valid compression algorithm id, otherwise returns false
+*/
+extern bool zs_is_valid_impl_id(unsigned int id);
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 00000000000..fe9eaa0f95f
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,117 @@
+/*
+ * zpq_stream.h
+ * Streaming compression for libpq
+ */
+#include "z_stream.h"
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#define ZPQ_DEFAULT_COMPRESSION_LEVEL (1)
+#define ZPQ_INCOMPLETE_HEADER (-6)
+#define ZPQ_FATAL_ERROR (-7)
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t (*zpq_tx_func) (void *arg, void const *data, size_t size);
+typedef ssize_t (*zpq_rx_func) (void *arg, void *data, size_t size);
+
+/*
+ * Descriptor of compression algorithm chosen by client
+ */
+typedef struct zpq_compressor
+{
+ unsigned int impl; /* compression algorithm index */
+ int level; /* compression level */
+} zpq_compressor;
+
+#endif
+
+/*
+ * Create compression stream with rx/tx function for reading/sending compressed data.
+ * tx_func: function for writing compressed data in underlying stream
+ * rx_func: function for reading compressed data from underlying stream
+ * arg: context passed to the function
+ * rx_data: received data (compressed data already fetched from input stream)
+ * rx_data_size: size of data fetched from input stream
+ */
+extern ZpqStream * zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size);
+
+/*
+ * Write up to "src_size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZPQ_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *src_processed.
+ */
+extern ssize_t zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed);
+
+/*
+ * Read up to "dst_size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZPQ_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zpq_read(ZpqStream * zpq, void *dst, size_t dst_size);
+
+/*
+ * Return true if non-flushed data left in internal rx decompression buffer.
+ */
+extern bool zpq_buffered_rx(ZpqStream * zpq);
+
+/*
+ * Return true if non-flushed data left in internal tx compression buffer.
+ */
+extern bool zpq_buffered_tx(ZpqStream * zpq);
+
+/*
+ * Free stream created by zs_create function.
+ */
+extern void zpq_free(ZpqStream * zpq);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zpq_decompress_error(ZpqStream * zpq);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zpq_compress_error(ZpqStream * zpq);
+
+/*
+ * Get the name of the current compression algorithm.
+ */
+extern char const *zpq_compress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Get the name of the current decompression algorithm.
+ */
+extern char const *zpq_decompress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Parse comma-separated list of compression algorithms. It can be either explicitly specified, or include all algorithms supported by client library.
+ * Returns:
+ * - true if the compression string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or if compression is disabled, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+extern bool
+ zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Serialize the compressors array to string so it can be transmitted to the other side during the compression startup.
+ * For example, for array of two compressors (zstd, level 1), (zlib, level 2) resulting string would look like "zstd:1,zlib:2".
+ * Returns the resulting string.
+ */
+extern char
+ *zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors);
+
+/* Deserialize the compressors string received during the compression setup to a compressors array.
+ * For example, for string "zstd:1,zlib:2" compressors would be populated with 2 elements: (zstd, level 1), (zlib, level 2).
+ * Returns:
+ * - true if the compressors string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or string is empty, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+bool
+ zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors);
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 02015efe13c..bc3e47b51f9 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -188,6 +188,9 @@ typedef struct Port
int keepalives_count;
int tcp_user_timeout;
+ char *compression_algorithms; /* Compression algorithms supported by
+ * client */
+
/*
* GSSAPI structures.
*/
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 6b67a2a3187..794dbb49113 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -71,6 +71,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port *port);
extern int pq_getbytes(char *s, size_t len);
extern void pq_startmsgread(void);
extern void pq_endmsgread(void);
diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h
index 1bcc189deeb..c46f5e00d99 100644
--- a/src/include/libpq/pqcomm.h
+++ b/src/include/libpq/pqcomm.h
@@ -137,6 +137,9 @@ typedef uint32 PacketLen;
extern bool Db_user_namespace;
+/* List of allowed compression algorithms */
+extern char *libpq_compress_algorithms;
+
/*
* In protocol 3.0 and later, the startup packet length is not fixed, but
* we set an arbitrary limit on it anyway. This is just to prevent simple
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index dd983fc9496..3d6adb467a3 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -181,6 +181,12 @@ typedef struct PgBackendStatus
/* application name; MUST be null-terminated */
char *st_appname;
+ /* client-server traffic information */
+ uint64 st_rx_raw_bytes;
+ uint64 st_tx_raw_bytes;
+ uint64 st_rx_compressed_bytes;
+ uint64 st_tx_compressed_bytes;
+
/*
* Current command string; MUST be null-terminated. Note that this string
* possibly is truncated in the middle of a multi-byte character. As
@@ -381,6 +387,7 @@ extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
+extern void pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes);
extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
int buflen);
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 40a96c881d6..21c29c0acc2 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,20 @@ endif
# The MSVC build system scrapes OBJS from this file. If you change any of
# the conditional additions of files to OBJS, update Mkvcbuild.pm to match.
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+SHLIB_LINK += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
+# We can't use Makefile variables here because the MSVC build system scrapes
+# OBJS from this file.
+
+
OBJS = \
$(WIN32RES) \
fe-auth-scram.o \
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index e8bcc883709..33ce6f2513f 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -186,3 +186,6 @@ PQpipelineStatus 183
PQsetTraceFlags 184
PQmblenBounded 185
PQsendFlushRequest 186
+PQcompressor 187
+PQdecompressor 188
+PQreadPending 189
\ No newline at end of file
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 9b6a6939f07..e121d4e71a2 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -24,6 +24,7 @@
#include "common/ip.h"
#include "common/link-canary.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "common/string.h"
#include "fe-auth.h"
#include "libpq-fe.h"
@@ -339,6 +340,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "PGCOMPRESSION", NULL, NULL,
+ "Libpq-compression", "", 16,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 15, /* sizeof("prefer-standby") = 15 */
@@ -447,6 +452,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zpqStream);
+ conn->zpqStream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -3271,11 +3280,14 @@ keep_going: /* We will come back to here until there is
}
/*
- * Validate message type: we expect only an authentication
- * request or an error here. Anything else probably means
- * it's not Postgres on the other end at all.
+ * Validate message type. We expect only:
+ * - authentication request ('R')
+ * - error ('E')
+ * - protocol compression acknowledgment ('z')
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
{
appendPQExpBuffer(&conn->errorMessage,
libpq_gettext("expected authentication request from server, but received %c\n"),
@@ -3353,6 +3365,60 @@ keep_going: /* We will come back to here until there is
return PGRES_POLLING_READING;
}
+ if (beresp == 'z') /* Switch on compression */
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ char *resp = malloc(msgLength);
+
+ pqGetnchar(resp, msgLength, conn);
+
+ if (!zpq_deserialize_compressors(resp, &compressors, &n_compressors))
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server returned unrecognized compression setting: %s\n"),
+ resp);
+ free(resp);
+ goto error_return;
+ }
+ free(resp);
+
+ if (n_compressors == 0)
+ {
+ /*
+ * If there are no compressors returned, it means that
+ * the server rejected all sent compression
+ * algorithms. Mark the data as consumed and do not
+ * enable the compression.
+ */
+ conn->inStart = conn->inCursor;
+ goto keep_going;
+ }
+
+ Assert(!conn->zpqStream);
+ conn->zpqStream = zpq_create(compressors, n_compressors,
+ (zpq_tx_func) pqsecure_write, (zpq_rx_func) pqsecure_read,
+ conn,
+ &conn->inBuffer[conn->inCursor],
+ conn->inEnd - conn->inCursor);
+ if (!conn->zpqStream)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("failed to initialize compression\n"));
+ free(compressors);
+ goto error_return;
+ }
+ /* reset buffer */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+
+ if (conn->n_compressors != 0 && beresp == 'v') /* NegotiateProtocolVersion */
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server does not support protocol compression\n"));
+ goto error_return;
+ }
+
/* Handle errors. */
if (beresp == 'E')
{
@@ -4097,6 +4163,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
@@ -6628,6 +6696,22 @@ PQuser(const PGconn *conn)
return conn->pguser;
}
+char *
+PQcompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_compress_algorithm_name(conn->zpqStream);
+}
+
+char *
+PQdecompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_decompress_algorithm_name(conn->zpqStream);
+}
+
char *
PQpass(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 6c7b3df0127..2061862688c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2010,9 +2010,7 @@ PQgetResult(PGconn *conn)
* EOF indication. We expect therefore that this won't result in any
* undue delay in reporting a previous write failure.)
*/
- if (flushResult ||
- pqWait(true, false, conn) ||
- pqReadData(conn) < 0)
+ if (flushResult || pqWait(true, false, conn) || pqReadData(conn) < 0)
{
/*
* conn->errorMessage has been set by pqWait or pqReadData. We
@@ -3785,6 +3783,12 @@ pqPipelineFlush(PGconn *conn)
return 0;
}
+int
+PQreadPending(PGconn *conn)
+{
+ return pqReadPending(conn);
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 9a2a9702934..8b7ab3367e5 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,12 +53,24 @@
#include "pg_config_paths.h"
#include "port/pg_bswap.h"
+#include <common/zpq_stream.h>
+
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
+/*
+ * Use zpq_read if compression is switched on
+ */
+#define pq_read_conn(conn) \
+ (conn->zpqStream \
+ ? zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd) \
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd))
+
/*
* PQlibVersion: return the libpq version number
*/
@@ -617,10 +629,17 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -712,10 +731,18 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
+
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -826,12 +853,18 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered_tx(conn->zpqStream))
{
int sent;
+ size_t processed = 0;
+ /*
+ * Use zpq_write if compression is switched on
+ */
+ sent = conn->zpqStream
+ ? zpq_write(conn->zpqStream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -839,8 +872,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+: pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +932,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered_tx(conn->zpqStream))
{
/*
* We didn't send it all, wait till we can send more.
@@ -989,6 +1025,8 @@ pqFlush(PGconn *conn)
int
pqWait(int forRead, int forWrite, PGconn *conn)
{
+ if (forRead && conn->inCursor < conn->inEnd)
+ return 0;
return pqWaitTimed(forRead, forWrite, conn, (time_t) -1);
}
@@ -1046,6 +1084,9 @@ pqWriteReady(PGconn *conn)
*
* If SSL is in use, the SSL buffer is checked prior to checking the socket
* for read data directly.
+ *
+ * If ZPQ stream is in use, the ZPQ buffer is checked prior to checking
+ * the socket for read data directly.
*/
static int
pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
@@ -1061,14 +1102,10 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return -1;
}
-#ifdef USE_SSL
- /* Check for SSL library buffering read bytes */
- if (forRead && conn->ssl_in_use && pgtls_read_pending(conn))
+ if (forRead && (pqReadPending(conn) > 0))
{
- /* short-circuit the select */
return 1;
}
-#endif
/* We will retry as long as we get EINTR */
do
@@ -1088,6 +1125,33 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return result;
}
+/*
+ * Check if there is some data pending in ZPQ / SSL read buffers.
+ * Returns -1 on failure, 0 if no, 1 if yes.
+ */
+int
+pqReadPending(PGconn *conn)
+{
+ if (!conn)
+ return -1;
+
+ /* check for ZPQ stream buffered read bytes */
+ if (zpq_buffered_rx(conn->zpqStream))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+
+#ifdef USE_SSL
+ /* Check for SSL library buffering read bytes */
+ if (conn->ssl_in_use && pgtls_read_pending(conn))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+#endif
+ return 0;
+}
/*
* Check a file descriptor for read and/or write data, possibly waiting.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 9ab3bf1fcb6..0619e1ab0d6 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -2174,6 +2174,50 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
return startpacket;
}
+/*
+ * Build comma-separated list of compression algorithms requested by client.
+ * It can be either explicitly specified by user in connection string, or
+ * include all algorithms supported by client library.
+ * This function returns true if the compression string is successfully parsed and
+ * stores a comma-separated list of algorithms in *client_compressors.
+ * If compression is disabled, then NULL is assigned to *client_compressors.
+ * Also it creates an array of compressor descriptors, each element of which corresponds to
+ * the corresponding algorithm name in *client_compressors list. This array is stored in PGconn
+ * and is used during handshake when a compression acknowledgment response is received from the server.
+ */
+static bool
+build_compressors_list(PGconn *conn, char **client_compressors, bool build_descriptors)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors))
+ {
+ return false;
+ }
+
+ *client_compressors = NULL;
+ if (build_descriptors)
+ {
+ conn->compressors = compressors;
+ conn->n_compressors = n_compressors;
+ }
+
+ if (n_compressors == 0)
+ {
+ /* no compressors available, return */
+ return true;
+ }
+
+ *client_compressors = zpq_serialize_compressors(compressors, n_compressors);
+
+ if (!build_descriptors)
+ {
+ free(compressors);
+ }
+ return true;
+}
+
/*
* Build a startup packet given a filled-in PGconn structure.
*
@@ -2220,6 +2264,18 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("replication", conn->replication);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
+ if (conn->compression && conn->compression[0])
+ {
+ char *client_compression_algorithms;
+
+ if (build_compressors_list((PGconn *) conn, &client_compression_algorithms, packet == NULL))
+ {
+ if (client_compression_algorithms)
+ {
+ ADD_STARTUP_OPTION("_pq_.compression", client_compression_algorithms);
+ }
+ }
+ }
if (conn->send_appname)
{
/* Use appname if present, otherwise use fallback */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index a6fd69acebc..1e951320d84 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -340,6 +340,8 @@ extern char *PQhostaddr(const PGconn *conn);
extern char *PQport(const PGconn *conn);
extern char *PQtty(const PGconn *conn);
extern char *PQoptions(const PGconn *conn);
+extern char *PQcompressor(const PGconn *conn);
+extern char *PQdecompressor(const PGconn *conn);
extern ConnStatusType PQstatus(const PGconn *conn);
extern PGTransactionStatusType PQtransactionStatus(const PGconn *conn);
extern const char *PQparameterStatus(const PGconn *conn,
@@ -498,6 +500,9 @@ extern PGPing PQpingParams(const char *const *keywords,
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
+extern int
+ PQreadPending(PGconn *conn);
+
/*
* "Fast path" interface --- not really recommended for application
* use
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 490458adef5..1e54a4c79ff 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "fe-auth-sasl.h"
#include "pqexpbuffer.h"
@@ -340,6 +341,7 @@ typedef struct pg_conn_host
* found in password file. */
} pg_conn_host;
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -392,6 +394,14 @@ struct pg_conn
* "sspi") */
char *ssl_min_protocol_version; /* minimum TLS protocol version */
char *ssl_max_protocol_version; /* maximum TLS protocol version */
+
+ char *compression; /* stream compression (boolean value, "any" or
+ * list of compression algorithms separated by
+ * comma) */
+ zpq_compressor *compressors; /* descriptors of compression algorithms
+ * chosen by client */
+ unsigned n_compressors; /* size of compressors array */
+
char *target_session_attrs; /* desired session properties */
/* Optional file to write trace info to */
@@ -572,6 +582,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream *zpqStream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
@@ -697,6 +710,7 @@ extern int pqWaitTimed(int forRead, int forWrite, PGconn *conn,
time_t finish_time);
extern int pqReadReady(PGconn *conn);
extern int pqWriteReady(PGconn *conn);
+extern int pqReadPending(PGconn *conn);
/* === in fe-secure.c === */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d635ada54de..4fb5695fa41 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1765,7 +1765,7 @@ pg_stat_activity| SELECT s.datid,
s.query_id,
s.query,
s.backend_type
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_all_indexes| SELECT c.oid AS relid,
@@ -1880,8 +1880,14 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_auth AS gss_authenticated,
s.gss_princ AS principal,
s.gss_enc AS encrypted
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
+pg_stat_network_traffic| SELECT s.pid,
+ s.rx_raw_bytes,
+ s.tx_raw_bytes,
+ s.rx_compressed_bytes,
+ s.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
@@ -2050,7 +2056,7 @@ pg_stat_replication| SELECT s.pid,
w.sync_priority,
w.sync_state,
w.reply_time
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_replication_slots| SELECT s.slot_name,
@@ -2084,7 +2090,7 @@ pg_stat_ssl| SELECT s.pid,
s.ssl_client_dn AS client_dn,
s.ssl_client_serial AS client_serial,
s.ssl_issuer_dn AS issuer_dn
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 404c45a6f30..c41370cd28d 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -130,7 +130,7 @@ sub mkvcbuild
keywords.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c
rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c
- username.c wait_error.c wchar.c);
+ username.c wait_error.c wchar.c z_stream.c zpq_stream.c);
if ($solution->{options}->{openssl})
{
--
2.17.1
0002-Add-ZSTD-support.patch (text/x-diff, 17.8K) download
From 6493d93ba86fd4b1155a85a3f7eb048e54d5c961 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:59:01 +0500
Subject: [PATCH 02/10] Add ZSTD support
---
configure | 214 +++++++++++++++++++++++++++++++
configure.ac | 36 ++++++
src/Makefile.global.in | 1 +
src/backend/Makefile | 4 +
src/common/z_stream.c | 228 ++++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
src/interfaces/libpq/Makefile | 5 +
src/tools/msvc/Solution.pm | 1 +
8 files changed, 492 insertions(+)
diff --git a/configure b/configure
index 3b19105328d..e49a16f8f4d 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,9 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -868,6 +871,7 @@ with_libxslt
with_system_tzdata
with_zlib
with_lz4
+with_zstd
with_gnu_ld
with_ssl
with_openssl
@@ -897,6 +901,8 @@ XML2_CFLAGS
XML2_LIBS
LZ4_CFLAGS
LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
LDFLAGS_EX
LDFLAGS_SL
PERL
@@ -1576,6 +1582,7 @@ Optional Packages:
use system time zone data in DIR
--without-zlib do not use Zlib
--with-lz4 build with LZ4 support
+ --with-zstd build with ZSTD support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -1605,6 +1612,8 @@ Some influential environment variables:
XML2_LIBS linker flags for XML2, overriding pkg-config
LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config
LZ4_LIBS linker flags for LZ4, overriding pkg-config
+ ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+ ZSTD_LIBS linker flags for ZSTD, overriding pkg-config
LDFLAGS_EX extra linker flags for linking executables only
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
@@ -9033,6 +9042,145 @@ fi
done
fi
+#
+# Zstd
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+
+$as_echo "#define HAVE_LIBZSTD 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd >= 1.0.0" >&5
+$as_echo_n "checking for libzstd >= 1.0.0... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+ pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+ pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ else
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$ZSTD_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (libzstd >= 1.0.0) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+ ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -12523,6 +12671,58 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_decompressStream in -lzstd" >&5
+$as_echo_n "checking for ZSTD_decompressStream in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_decompressStream+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_decompressStream ();
+int
+main ()
+{
+return ZSTD_decompressStream ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_decompressStream=yes
+else
+ ac_cv_lib_zstd_ZSTD_decompressStream=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_decompressStream" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_decompressStream" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_decompressStream" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13854,6 +14054,20 @@ fi
done
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index e77d4dcf2d2..3d554869ad8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,28 @@ if test "$with_lz4" = yes; then
done
fi
+#
+# Zstd
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support], [AC_DEFINE([HAVE_LIBZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+ PKG_CHECK_MODULES(ZSTD, [libzstd >= 1.0.0])
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -1242,6 +1264,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_decompressStream, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1487,6 +1516,13 @@ if test "$with_lz4" = yes; then
AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 79ed589ce94..71e14ee68f9 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -197,6 +197,7 @@ with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
with_lz4 = @with_lz4@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index f65f16fc480..2074d23d5d6 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,10 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
ifeq ($(with_lz4),yes)
LIBS += -llz4
endif
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index d94914e7dde..6f490ebe781 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -89,6 +89,231 @@ struct ZStream
bool not_flushed;
};
+#if HAVE_LIBZSTD
+
+#include <stdlib.h>
+#include <zstd.h>
+
+/*
+ * Maximum allowed back-reference distance, expressed as power of 2.
+ * This setting controls max compressor/decompressor window size.
+ * More details https://github.com/facebook/zstd/blob/v1.4.7/lib/zstd.h#L536
+ */
+#define ZSTD_WINDOWLOG_LIMIT 23 /* set max window size to 8MB */
+
+
+typedef struct ZS_ZSTD_CStream
+{
+ ZSTD_CStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_CStream;
+
+typedef struct ZS_ZSTD_DStream
+{
+ ZSTD_DStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_DStream;
+
+static void *
+zstd_create_compressor(int level)
+{
+ size_t rc;
+ ZS_ZSTD_CStream *c_stream = (ZS_ZSTD_CStream *) malloc(sizeof(ZS_ZSTD_CStream));
+
+ c_stream->stream = ZSTD_createCStream();
+ rc = ZSTD_initCStream(c_stream->stream, level);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeCStream(c_stream->stream);
+ free(c_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_CCtx_setParameter(c_stream->stream, ZSTD_c_windowLog, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ c_stream->error = NULL;
+ return c_stream;
+}
+
+static void *
+zstd_create_decompressor()
+{
+ size_t rc;
+ ZS_ZSTD_DStream *d_stream = (ZS_ZSTD_DStream *) malloc(sizeof(ZS_ZSTD_DStream));
+
+ d_stream->stream = ZSTD_createDStream();
+ rc = ZSTD_initDStream(d_stream->stream);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeDStream(d_stream->stream);
+ free(d_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_DCtx_setParameter(d_stream->stream, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ d_stream->error = NULL;
+ return d_stream;
+}
+
+static ssize_t
+zstd_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+ size_t rc;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ rc = ZSTD_decompressStream(ds->stream, &out, &in);
+
+ *src_processed = in.pos;
+ *dst_processed = out.pos;
+ if (ZSTD_isError(rc))
+ {
+ ds->error = ZSTD_getErrorName(rc);
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ if (rc == 0)
+ {
+ return ZS_STREAM_END;
+ }
+
+ if (out.pos == out.size)
+ {
+ /*
+ * if `output.pos == output.size`, there might be some data left
+ * within internal buffers
+ */
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ if (in.pos < src_size) /* Has something to compress in input buffer */
+ {
+ size_t rc = ZSTD_compressStream(cs->stream, &out, &in);
+
+ *dst_processed = out.pos;
+ *src_processed = in.pos;
+ if (ZSTD_isError(rc))
+ {
+ cs->error = ZSTD_getErrorName(rc);
+ return ZS_COMPRESS_ERROR;
+ }
+ }
+
+ if (in.pos == src_size) /* All data is compressed: flush internal zstd
+ * buffer */
+ {
+ size_t tx_not_flushed = ZSTD_flushStream(cs->stream, &out);
+
+ *dst_processed = out.pos;
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ size_t tx_not_flushed;
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_outBuffer output;
+
+ output.dst = dst;
+ output.pos = 0;
+ output.size = dst_size;
+
+ do
+ {
+ tx_not_flushed = ZSTD_endStream(cs->stream, &output);
+ } while ((tx_not_flushed > 0) && (output.pos < output.size));
+
+ *dst_processed = output.pos;
+
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static void
+zstd_free_compressor(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ ZSTD_freeCStream(cs->stream);
+ free(cs);
+ }
+}
+
+static void
+zstd_free_decompressor(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ ZSTD_freeDStream(ds->stream);
+ free(ds);
+ }
+}
+
+static char const *
+zstd_compress_error(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ return cs->error;
+}
+
+static char const *
+zstd_decompress_error(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ return ds->error;
+}
+
+static char const *
+zstd_name(void)
+{
+ return "zstd";
+}
+
+#endif
+
#if HAVE_LIBZ
#include <stdlib.h>
@@ -421,6 +646,9 @@ no_compression_name(void)
*/
static ZAlgorithm const zs_algorithms[] =
{
+#if HAVE_LIBZSTD
+ {zstd_name, zstd_create_compressor, zstd_create_decompressor, zstd_decompress, zstd_compress, zstd_free_compressor, zstd_free_decompressor, zstd_compress_error, zstd_decompress_error, zstd_end},
+#endif
#if HAVE_LIBZ
{zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
#endif
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 7525c165974..a63685a6392 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 21c29c0acc2..85fda6b2363 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -34,6 +34,11 @@ LIBS += -llz4
SHLIB_LINK += -llz4
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
ifeq ($(with_zlib),yes)
LIBS += -lz
SHLIB_LINK += -lz
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index 2c8cd521e94..5ecd0a1832a 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
--
2.17.1
0003-Avoid-buffer-overflow-found-by-valgrind.patch (text/x-diff, 1.2K) download
From e85378b2245e9b9d02bf9c4c307795f026d0ab2b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 31 Dec 2021 15:04:40 -0600
Subject: [PATCH 03/10] Avoid buffer overflow found by valgrind
---
src/common/zpq_stream.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
index 294ad249ce7..307444e846d 100644
--- a/src/common/zpq_stream.c
+++ b/src/common/zpq_stream.c
@@ -985,7 +985,7 @@ zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressor
/*
* single entry looks like "alg_name:compression_level," so +2 is for
- * ":" and "," symbols
+ * ":" and "," symbols (or trailing null)
*/
total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
}
@@ -994,8 +994,9 @@ zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressor
for (i = 0; i < n_compressors; i++)
{
- p += sprintf(p, "%s:%d,", supported_algorithms[compressors[i].impl], compressors[i].level);
+ p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
+ if (i < n_compressors - 1)
+ *p++ = ',';
}
- p[-1] = '\0';
return res;
}
--
2.17.1
0004-fix-compiler-warnings.patch (text/x-diff, 2.2K) download
From a5b3fe49e05bfe7303e5fa75687b845a4c92efb1 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 31 Dec 2021 13:49:09 -0600
Subject: [PATCH 04/10] fix compiler warnings
---
src/common/z_stream.c | 18 +++++++-----------
1 file changed, 7 insertions(+), 11 deletions(-)
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index 6f490ebe781..ace88ad566b 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -383,7 +383,7 @@ static ssize_t
zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
{
z_stream *cs = (z_stream *) c_stream;
- int rc;
+ int rc PG_USED_FOR_ASSERTS_ONLY;
unsigned deflate_pending = 0;
@@ -530,15 +530,12 @@ lz4_create_decompressor()
static ssize_t
lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
{
- ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
- Assert(src_size > 0);
-
- char* const decPtr = &ds->buf[ds->buf_pos];
-
- const int decBytes = LZ4_decompress_safe_continue(
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+ char *const decPtr = &ds->buf[ds->buf_pos];
+ const int decBytes = LZ4_decompress_safe_continue(
ds->stream, src, decPtr, (int) src_size, (int) dst_size);
- Assert(decBytes > 0);
+ Assert(decBytes > 0);
*dst_processed = decBytes;
*src_processed = src_size;
@@ -558,16 +555,15 @@ static ssize_t
lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
{
ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+ int cmpBytes;
src_size = Min(MESSAGE_MAX_BYTES, src_size);
- Assert(src_size > 0);
-
memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
Assert(dst_size >= LZ4_compressBound(src_size));
- const int cmpBytes = LZ4_compress_fast_continue(
+ cmpBytes = LZ4_compress_fast_continue(
cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, 1);
Assert(cmpBytes > 0);
--
2.17.1
0005-respect-LZ4-compression-level.patch (text/x-diff, 931B) download
From 52a099230a79bd49854f25f2f2803a8c87f9ff2d Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 31 Dec 2021 22:07:17 -0600
Subject: [PATCH 05/10] respect LZ4 compression level
---
src/common/z_stream.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index ace88ad566b..9bc28f85eab 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -564,7 +564,7 @@ lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_proce
Assert(dst_size >= LZ4_compressBound(src_size));
cmpBytes = LZ4_compress_fast_continue(
- cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, 1);
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, cs->level);
Assert(cmpBytes > 0);
Assert(cmpBytes <= MESSAGE_MAX_BYTES);
--
2.17.1
0006-zpq_choose_compressor.patch (text/x-diff, 769B) download
From e3cdeaab82d34b7a8e338626de9ca837bc4a343e Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 31 Dec 2021 22:53:58 -0600
Subject: [PATCH 06/10] zpq_choose_compressor
---
src/common/zpq_stream.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
index 307444e846d..bd94b1156fa 100644
--- a/src/common/zpq_stream.c
+++ b/src/common/zpq_stream.c
@@ -197,7 +197,7 @@ zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
static inline bool
zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
{
- return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
+ return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
}
/*
--
2.17.1
0007-conninfo-on-one-line.patch (text/x-diff, 921B) download
From 1f3f985bc28fad9ecfac8fd5fd83afc7378769c3 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 1 Jan 2022 01:25:52 -0600
Subject: [PATCH 07/10] conninfo on one line
---
src/bin/psql/command.c | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index c87d86db58e..55c78e99382 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -3660,15 +3660,9 @@ printCompressionInfo(void)
char *compressor = PQcompressor(pset.db);
char *decompressor = PQdecompressor(pset.db);
- if (compressor != NULL)
- {
- printf(_("Compressor %s\n"), compressor);
- }
-
- if (decompressor != NULL)
- {
- printf(_("Decompressor %s\n"), decompressor);
- }
+ printf(_("Compression: client: %s, server: %s\n"),
+ compressor ? compressor : "none",
+ decompressor ? decompressor : "none");
}
/*
--
2.17.1
0008-Enable-zlib-compression-by-default.patch (text/x-diff, 864B) download
From c75634402bc7642b096d8172b52566b8a44b6e96 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Fri, 19 Mar 2021 00:54:11 -0500
Subject: [PATCH 08/10] Enable zlib compression by default
For CI - not for merge
---
src/interfaces/libpq/fe-connect.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index e121d4e71a2..9756aa29d52 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -340,7 +340,7 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
- {"compression", "PGCOMPRESSION", NULL, NULL,
+ {"compression", "PGCOMPRESSION", "zlib", NULL,
"Libpq-compression", "", 16,
offsetof(struct pg_conn, compression)},
--
2.17.1
* Re: libpq compression (part 2)
@ 2022-01-03 11:16 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 0 replies; 27+ messages in thread
From: Andrey Borodin @ 2022-01-03 11:16 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Daniil Zakhlystov <usernamedt@yandex-team.ru>
> Maybe you should reset the streams between each compression message (even if
> it's using the same compression algorithm). This might allow better
> compression.
AFAIK on the contrary - longer data sequence usually compresses better. The codec can use knowledge about prior data to better compress current bytes.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-07 18:46 Bruce Momjian <bruce@momjian.us>
parent: Justin Pryzby <pryzby@telsasoft.com>
1 sibling, 1 reply; 27+ messages in thread
From: Bruce Momjian @ 2022-01-07 18:46 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> Thanks for working on this. The patch looks to be in good shape - I hope more
> people will help to review and test it. I took the liberty of creating a new
> CF entry. http://cfbot.cputube.org/daniil-zakhlystov.html
>
> +zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
> +{
> + return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
>
> I think this is backwards , and should say != -1 ?
>
> As written, the server GUC libpq_compression defaults to "on", and the client
> doesn't request compression. I think the server GUC should default to off.
> I failed to convince Kontantin about this last year. The reason is that 1)
> it's a new feature; 2) with security implications. An admin should need to
> "opt in" to this. I still wonder if this should be controlled by a new "TYPE"
> in pg_hba (rather than a GUC); that would make it exclusive of SSL.
I assume this compression happens before it is encrypted for TLS
transport. Second, compression was removed from TLS because there were
too many ways for HTTP to weaken encryption. I assume the Postgres wire
protocol doesn't have similar exploit possibilities.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-07 19:21 Justin Pryzby <pryzby@telsasoft.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-01-07 19:21 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
On Fri, Jan 07, 2022 at 01:46:00PM -0500, Bruce Momjian wrote:
> On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> > Thanks for working on this. The patch looks to be in good shape - I hope more
> > people will help to review and test it. I took the liberty of creating a new
> > CF entry. http://cfbot.cputube.org/daniil-zakhlystov.html
> >
> > +zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
> > +{
> > + return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
> >
> > I think this is backwards , and should say != -1 ?
> >
> > As written, the server GUC libpq_compression defaults to "on", and the client
> > doesn't request compression. I think the server GUC should default to off.
> > I failed to convince Kontantin about this last year. The reason is that 1)
> > it's a new feature; 2) with security implications. An admin should need to
> > "opt in" to this. I still wonder if this should be controlled by a new "TYPE"
> > in pg_hba (rather than a GUC); that would make it exclusive of SSL.
>
> I assume this compression happens before it is encrypted for TLS
> transport. Second, compression was removed from TLS because there were
> too many ways for HTTP to weaken encryption. I assume the Postgres wire
> protocol doesn't have similar exploit possibilities.
It's discussed in last year's thread. The thinking is that there tends to be
*fewer* exploitable opportunities between application->DB than between
browser->app.
But it's still a known concern, and should default to off - as I said.
That's also why I wondered if compression should be controlled by pg_hba,
rather than a GUC. To require/allow an DBA to opt-in to it for specific hosts.
Or to make it exclusive of ssl. We could choose to not suppose that case at
all, or (depending on the implement) refuse that combination of layers.
--
Justin
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-07 20:56 Stephen Frost <sfrost@snowman.net>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 2 replies; 27+ messages in thread
From: Stephen Frost @ 2022-01-07 20:56 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
Greetings,
* Justin Pryzby (pryzby@telsasoft.com) wrote:
> On Fri, Jan 07, 2022 at 01:46:00PM -0500, Bruce Momjian wrote:
> > On Sat, Jan 1, 2022 at 11:25:05AM -0600, Justin Pryzby wrote:
> > > Thanks for working on this. The patch looks to be in good shape - I hope more
> > > people will help to review and test it. I took the liberty of creating a new
> > > CF entry. http://cfbot.cputube.org/daniil-zakhlystov.html
> > >
> > > +zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
> > > +{
> > > + return zpq_choose_compressor(zpq, msg_type, msg_len) == -1;
> > >
> > > I think this is backwards , and should say != -1 ?
> > >
> > > As written, the server GUC libpq_compression defaults to "on", and the client
> > > doesn't request compression. I think the server GUC should default to off.
> > > I failed to convince Kontantin about this last year. The reason is that 1)
> > > it's a new feature; 2) with security implications. An admin should need to
> > > "opt in" to this. I still wonder if this should be controlled by a new "TYPE"
> > > in pg_hba (rather than a GUC); that would make it exclusive of SSL.
> >
> > I assume this compression happens before it is encrypted for TLS
> > transport. Second, compression was removed from TLS because there were
> > too many ways for HTTP to weaken encryption. I assume the Postgres wire
> > protocol doesn't have similar exploit possibilities.
>
> It's discussed in last year's thread. The thinking is that there tends to be
> *fewer* exploitable opportunities between application->DB than between
> browser->app.
Yes, this was discussed previously and addressed.
> But it's still a known concern, and should default to off - as I said.
I'm not entirely convinced of this but also am happy enough as long as
the capability exists, no matter if it's off or on by default.
> That's also why I wondered if compression should be controlled by pg_hba,
> rather than a GUC. To require/allow an DBA to opt-in to it for specific hosts.
> Or to make it exclusive of ssl. We could choose to not suppose that case at
> all, or (depending on the implement) refuse that combination of layers.
I'm definitely against us deciding that we know better than admins if
this is an acceptable trade-off in their environment, or not. Letting
users/admins control it is fine, but I don't feel we should forbid it.
As for the details of how we allow control over it, I suppose there's a
number of options. Having it in the HBA doesn't seem terrible, though I
suspect most will just want to enable it across the board and having to
have "compression=allowed" or whatever added to every hba line seems
likely to be annoying. Maybe a global GUC and then allow the hba to
override?
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-07 21:09 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 27+ messages in thread
From: Bruce Momjian @ 2022-01-07 21:09 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
On Fri, Jan 7, 2022 at 03:56:39PM -0500, Stephen Frost wrote:
> As for the details of how we allow control over it, I suppose there's a
> number of options. Having it in the HBA doesn't seem terrible, though I
> suspect most will just want to enable it across the board and having to
> have "compression=allowed" or whatever added to every hba line seems
> likely to be annoying. Maybe a global GUC and then allow the hba to
> override?
Ewe, I would like to avoid going in the GUC & pg_hba.conf direction,
unless we have other cases where we already do this.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-07 21:17 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 27+ messages in thread
From: Stephen Frost @ 2022-01-07 21:17 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Jan 7, 2022 at 03:56:39PM -0500, Stephen Frost wrote:
> > As for the details of how we allow control over it, I suppose there's a
> > number of options. Having it in the HBA doesn't seem terrible, though I
> > suspect most will just want to enable it across the board and having to
> > have "compression=allowed" or whatever added to every hba line seems
> > likely to be annoying. Maybe a global GUC and then allow the hba to
> > override?
>
> Ewe, I would like to avoid going in the GUC & pg_hba.conf direction,
> unless we have other cases where we already do this.
I mean, not exactly the same, but ... ssl?
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-12 09:38 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 0 replies; 27+ messages in thread
From: Andrey Borodin @ 2022-01-12 09:38 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Bruce Momjian <bruce@momjian.us>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
> 8 янв. 2022 г., в 01:56, Stephen Frost <sfrost@snowman.net> написал(а):
>>
>> It's discussed in last year's thread. The thinking is that there tends to be
>> *fewer* exploitable opportunities between application->DB than between
>> browser->app.
>
> Yes, this was discussed previously and addressed.
What else do we need to decide architecturally to make protocol compression happen in 15? As far as I can see - only HBA\GUC part.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-12 14:15 Justin Pryzby <pryzby@telsasoft.com>
parent: Daniil Zakhlystov <usernamedt@yandex-team.ru>
1 sibling, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-01-12 14:15 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; +Cc: pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
zlib still causes check-world to get stuck. I first mentioned this last March:
20210319062800.GI11765@telsasoft.com
Actually all the compression methods seems to get stuck with
time make check -C src/bin/pg_rewind
time make check -C src/test/isolation
For CI purposes, there should be an 0003 patch which enables compression by
default, for all message types and maybe all lengths.
I removed the thread from the CFBOT until that's resolved.
--
Justin
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-13 21:12 Daniil Zakhlystov <usernamedt@yandex-team.ru>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 27+ messages in thread
From: Daniil Zakhlystov @ 2022-01-13 21:12 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
Hi, Justin!
First of all, thanks for the detailed review. I’ve applied your patches to the current version.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> I wonder whether the asymmetric compression idea is useful. The only
> application I can see for this is that a server might allow to *decompress*
> data from a client, but may not want to *compress* data to a client.
In the current patch state, there is no option to forbid the server-to-client or client-to-server
compression only. The decision of whether to compress or not is based only on the message length. In
the future, we can implement more precise control without the need for any protocol changes.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> I'm not sure it's needed to allow the compression to be changed dynamically,
> and the generality to support a different compression mthod for each libpq
> message type seems excessive. Maybe it's enough to check that the message type
> is one of VALID_LONG_MESSAGE_TYPE and its length is long enough.
Yes, as stated above, in the current patch version protocol message type is ignored but can be potentially
be taken into account. All messages will be compressed using the first available compressor. For
example, if postgresql.conf contains “libpq_compression = zlib,zstd” and has "compression = zlib,zstd”
in its connection string, zlib will be used to compress all messages with length more than the threshold.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> What happenes if the compression doesn't decrease the message size ? I don't
> see anything that allows sending the original, raw data. The advantage would
> be that the remote side doesn't incur the overhead of decompression.
Because of the streaming compression, we can’t just throw away some compressed data because future
compressed messages might back-reference it. There are two ways to solve this:
1. Do not use the streaming compression
2. Somehow reset the compression context to the state before the message has been compressed
I’ll look into it. Also, if anyone has any thoughts on this, I’ll appreciate hearing your opinions.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> I hit this assertion with PGCOMPRESSION=lz4 (but not zlib), and haven't yet
> found the cause.
Thanks for reporting this. Looks like LZ4 implementation is not polished yet and needs some
additional effort. I'll look into it too.
> On 12 Jan 2022, at 19:15, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> zlib still causes check-world to get stuck. I first mentioned this last March:
> 20210319062800.GI11765@telsasoft.com
>
> Actually all the compression methods seems to get stuck with
> time make check -C src/bin/pg_rewind
> time make check -C src/test/isolation
>
> For CI purposes, there should be an 0003 patch which enables compression by
> default, for all message types and maybe all lengths.
>
> I removed the thread from the CFBOT until that's resolved.
I’ve fixed the failing tests, now they should pass.
Thanks,
Daniil Zakhlystov
Attachments: download all (3 files, .tar.gz)
0001-Implement-libpq-compression.patch (application/octet-stream, 115.6K) download
From 60d2908baa01e95bdcda1c6ad1b9e492733bba6f Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Fri, 14 Jan 2022 01:38:55 +0500
Subject: [PATCH 1/3] Implement libpq compression
---
.../postgres_fdw/expected/postgres_fdw.out | 2 +-
doc/src/sgml/config.sgml | 21 +
doc/src/sgml/libpq.sgml | 37 +
doc/src/sgml/protocol.sgml | 174 +++
src/Makefile.global.in | 1 +
src/backend/Makefile | 8 +
src/backend/catalog/system_views.sql | 9 +
src/backend/libpq/pqcomm.c | 277 ++++-
src/backend/postmaster/postmaster.c | 10 +
src/backend/utils/activity/backend_status.c | 30 +
src/backend/utils/adt/pgstatfuncs.c | 50 +-
src/backend/utils/misc/guc.c | 30 +
src/bin/pgbench/pgbench.c | 17 +-
src/bin/psql/command.c | 17 +
src/common/Makefile | 4 +-
src/common/z_stream.c | 655 +++++++++++
src/common/zpq_stream.c | 1007 +++++++++++++++++
src/include/catalog/pg_proc.dat | 18 +-
src/include/common/z_stream.h | 107 ++
src/include/common/zpq_stream.h | 117 ++
src/include/libpq/libpq-be.h | 3 +
src/include/libpq/libpq.h | 1 +
src/include/libpq/pqcomm.h | 3 +
src/include/utils/backend_status.h | 7 +
src/interfaces/libpq/Makefile | 14 +
src/interfaces/libpq/exports.txt | 3 +
src/interfaces/libpq/fe-connect.c | 92 +-
src/interfaces/libpq/fe-exec.c | 10 +-
src/interfaces/libpq/fe-misc.c | 92 +-
src/interfaces/libpq/fe-protocol3.c | 67 +-
src/interfaces/libpq/libpq-fe.h | 5 +
src/interfaces/libpq/libpq-int.h | 14 +
src/test/regress/expected/rules.out | 14 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
34 files changed, 2827 insertions(+), 91 deletions(-)
create mode 100644 src/common/z_stream.c
create mode 100644 src/common/zpq_stream.c
create mode 100644 src/include/common/z_stream.h
create mode 100644 src/include/common/zpq_stream.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7d6f7d9e3d..0d30cfc4aa 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9509,7 +9509,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
+HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, compression, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index afbb6c35e3..8840edf913 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1035,6 +1035,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-libpq-compression" xreflabel="libpq_compression">
+ <term><varname>libpq_compression</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>libpq_compression</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter controls the available client-server traffic compression methods.
+ It allows rejecting compression requests even if it is supported by the server (for example, due to security, or CPU consumption).
+ The default is <literal>on</literal>, which means that all supported compression methods are allowed.
+ For more precise control, a list of the allowed compression methods can be specified.
+ For example, to allow only <literal>lz4</literal> and <literal>zlib</literal>, set the setting to <literal>lz4,zlib</literal>.
+ Also, maximal allowed compression level can be specified for each method, e.g. <literal>lz4:1,zlib:2</literal> setting will set the
+ maximal compression level for <literal>lz4</literal> to 1 and <literal>zlib</literal> to 2.
+ If a client requests the compression with a higher compression level, it will be set to the maximal allowed one.
+ Default (and recommended) maximal compression level for each algorithm is 1.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 14f35d37f6..5a16b4b64b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1260,6 +1260,43 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. The client sends a request with
+ a list of compression algorithms. Compression can be requested by a client
+ by including the <literal>"compression"</literal> option in its connection string.
+ This can either be a boolean value to enable or disable compression
+ (<literal>"true"</literal>/<literal>"false"</literal>,
+ <literal>"on"</literal>/<literal>"off"</literal>,
+ <literal>"yes"</literal>/<literal>"no"</literal>,
+ <literal>"1"</literal>/<literal>"0"</literal>),
+ <literal>"any"</literal>,
+ or an explicit list of comma-separated compression algorithms
+ which can optionally include compression level (<literal>"zlib,lz4:2"</literal>).
+ If compression is enabled but an algorithm is not explicitly specified,
+ the client library sends its full list of supported algorithms.
+ The server intersects the received compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ </para>
+ <para>
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ Compressed data is transmitted via the CompressedData messages.
+ </para>
+ <para>
+ Support for compression algorithms must be enabled when the server is compiled.
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option). In both cases, streaming mode is used.
+ By default, compression is not requested by the client.
+ Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..332041030a 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ It is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especially useful for importing/exporting data to/from the database using the <literal>COPY</literal> command
+ and for replication (both physical and logical). Compression can also reduce the server's response time
+ for queries returning a large amount of data (for example, JSON, BLOBs, text, ...).
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option).
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -262,6 +271,21 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CompressionAck</term>
+ <listitem>
+ <para>
+ The server accepts the client's compression request.
+ Compression is requested when a client connection includes the "compression" option, which contains a list of the requested compression algorithms.
+ The server intersects the requested compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>AuthenticationOk</term>
<listitem>
@@ -3577,6 +3601,141 @@ following:
</varlistentry>
+<varlistentry>
+<term>
+CompressionAck (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ List of the negotiated compression algorithms.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
+<varlistentry>
+<term>
+SetCompressionMethod (F & B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('k')
+</term>
+<listitem>
+<para>
+ Switch the current compression algorithm. Following CompressedData messages will be compressed via the compressed algorithm specified in this message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32(5)
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ Index of the new compression algorithm.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+CompressedData (F & B)
+</term>
+<listitem>
+<para>
+
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('m')
+</term>
+<listitem>
+<para>
+ Identifies the message as compressed data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ Compressed message data.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
AuthenticationSASLContinue (B)
@@ -6142,6 +6301,21 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>_pq_.compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. The value is a list of compression algorithms requested by the client with an optional
+ specification of compression level: <literal>"zlib,lz4:5"</literal>.
+ If the server does not support compression, the backend will ignore the _pq_.compression
+ parameter and will not send the CompressionAck message to the frontend.
+ By default, compression is disabled. Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 05c54b27de..79ed589ce9 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_lz4 = @with_lz4@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index add9560be4..dab379b412 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 701ff38f76..281abeebe8 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -844,6 +844,15 @@ CREATE VIEW pg_stat_activity AS
LEFT JOIN pg_database AS D ON (S.datid = D.oid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
+CREATE VIEW pg_stat_network_traffic AS
+ SELECT
+ S.pid,
+ S.rx_raw_bytes,
+ S.tx_raw_bytes,
+ S.rx_compressed_bytes,
+ S.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL) AS S;
+
CREATE VIEW pg_stat_replication AS
SELECT
S.pid,
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index f05723dc92..c200cd7ba0 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -60,6 +60,7 @@
#include <signal.h>
#include <fcntl.h>
#include <grp.h>
+#include <pgstat.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/socket.h>
@@ -77,11 +78,14 @@
#include "common/ip.h"
#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
#include "miscadmin.h"
#include "port/pg_bswap.h"
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/builtins.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -108,6 +112,9 @@
int Unix_socket_permissions;
char *Unix_socket_group;
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms;
+
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -130,6 +137,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream * PqStream;
+
+
/*
* Message status
*/
@@ -167,6 +177,145 @@ const PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+static ssize_t
+write_compressed(void *arg, void const *data, size_t size)
+{
+ ssize_t rc = secure_write((Port *) arg, (void *) data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, 0, rc);
+ return rc;
+}
+
+static ssize_t
+read_compressed(void *arg, void *data, size_t size)
+{
+ ssize_t rc = secure_read((Port *) arg, data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, rc, 0);
+ return rc;
+}
+
+/*
+ * Send the chosen compression algorithms to the client
+ */
+static void
+SendCompressionACK(char *compressors_str)
+{
+ StringInfoData buf;
+
+ pq_beginmessage(&buf, 'z');
+ pq_sendstring(&buf, compressors_str);
+ pq_endmessage(&buf);
+ pq_flush();
+}
+
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port *port)
+{
+ zpq_compressor *server_compressors;
+ zpq_compressor *client_compressors;
+ zpq_compressor *res_compressors;
+ size_t n_client_compressors;
+ size_t n_server_compressors;
+ size_t n_res_compressors = 0;
+ char *res_compressors_str;
+ char *client_compression_algorithms = port->compression_algorithms;
+
+ if (!client_compression_algorithms || !libpq_compress_algorithms)
+ {
+ return 0;
+ }
+
+ if (!zpq_parse_compression_setting(libpq_compress_algorithms, &server_compressors, &n_server_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms)));
+ return 0;
+ }
+
+ if (n_server_compressors == 0)
+ {
+ /*
+ * No enabled server compressors available, abort the compression
+ * initialization.
+ */
+ /* Send the compression acknowledgment with empty compressors list. */
+ char *empty_response = "";
+
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ if (!zpq_deserialize_compressors(client_compression_algorithms, &client_compressors, &n_client_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse received client compression methods: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ if (n_client_compressors == 0)
+ {
+ /*
+ * client did not provide any compression algorithms that server is
+ * compiled to support
+ */
+ ereport(LOG, (errmsg("server doesn't support any of the compression methods requested by the client: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ res_compressors = malloc(Max(n_client_compressors, n_server_compressors) * sizeof(zpq_compressor));
+
+ /*
+ * Intersect client and server compressors to determine the final list of
+ * the supported compressors. O(N^2) is negligible because of a small
+ * number of the compression methods.
+ */
+ for (size_t i = 0; i < n_client_compressors; i++)
+ {
+ for (size_t j = 0; j < n_server_compressors; j++)
+ {
+ if (client_compressors[i].impl == server_compressors[j].impl)
+ {
+ res_compressors[n_res_compressors].impl = client_compressors[i].impl;
+ /* prefer the lower compression level */
+ res_compressors[n_res_compressors++].level = Min(client_compressors[i].level, server_compressors[j].level);
+ break;
+ }
+ }
+ }
+ free(client_compressors);
+ free(server_compressors);
+
+ if (n_res_compressors == 0)
+ {
+ char *empty_response = "";
+
+ ereport(LOG, (errmsg("did not find any matches between the client and server compression methods, not enabling compression")));
+ free(res_compressors);
+ /* send the compression ack with empty compressors list */
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ res_compressors_str = zpq_serialize_compressors(res_compressors, n_res_compressors);
+ SendCompressionACK(res_compressors_str);
+
+ /* Init compression */
+ PqStream = zpq_create(res_compressors, n_res_compressors, write_compressed, read_compressed, MyProcPort, NULL, 0);
+ if (!PqStream)
+ {
+ ereport(LOG, (errmsg("failed to initialize the compression stream")));
+ return -1;
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -270,6 +419,9 @@ socket_close(int code, Datum arg)
}
#endif /* ENABLE_GSS */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -941,12 +1093,15 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * nowait parameter toggles non-blocking mode.
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -962,21 +1117,40 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ /*
+ * If streaming compression is enabled then use correspondent
+ * compression read function.
+ */
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, false)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
if (r < 0)
{
+ if (r == ZS_DECOMPRESS_ERROR)
+ {
+ char const *msg = zpq_decompress_error(PqStream);
+
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -997,7 +1171,8 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ pgstat_report_network_traffic(r, 0, 0, 0);
+ return r;
}
}
@@ -1012,7 +1187,8 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1031,7 +1207,8 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1048,48 +1225,15 @@ pq_peekbyte(void)
int
pq_getbyte_if_available(unsigned char *c)
{
- int r;
+ int r = 0;
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1110,7 +1254,8 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1144,7 +1289,8 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1168,7 +1314,6 @@ pq_buffer_has_data(void)
return (PqRecvPointer < PqRecvLength);
}
-
/* --------------------------------
* pq_startmsgread - begin reading a message from the client.
*
@@ -1372,13 +1517,24 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered_tx(PqStream) != 0)
+
+ /*
+ * has more data to flush or unsent data in internal compression
+ * buffer
+ */
{
int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
- if (r <= 0)
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1421,12 +1577,12 @@ internal_flush(void)
InterruptPending = 1;
return EOF;
}
+ pgstat_report_network_traffic(0, r, 0, 0);
last_reported_send_errno = 0; /* reset after any successful send */
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
@@ -1443,7 +1599,7 @@ socket_flush_if_writable(void)
int res;
/* Quick exit if nothing to do */
- if (PqSendPointer == PqSendStart)
+ if ((PqSendPointer == PqSendStart) && (zpq_buffered_tx(PqStream) == 0))
return 0;
/* No-op if reentrant call */
@@ -1466,7 +1622,7 @@ socket_flush_if_writable(void)
static bool
socket_is_send_pending(void)
{
- return (PqSendStart < PqSendPointer);
+ return (PqSendStart < PqSendPointer || (zpq_buffered_tx(PqStream) != 0));
}
/* --------------------------------
@@ -1989,3 +2145,16 @@ pq_check_connection(void)
return true;
}
+
+PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+
+Datum
+pg_compression_algorithm(PG_FUNCTION_ARGS)
+{
+ char const *algorithm_name = PqStream ? zpq_compress_algorithm_name(PqStream) : NULL;
+
+ if (algorithm_name)
+ PG_RETURN_TEXT_P(cstring_to_text(algorithm_name));
+ else
+ PG_RETURN_NULL();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index dc4afdd75a..39dedff751 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2233,6 +2233,8 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "_pq_.compression") == 0)
+ port->compression_algorithms = pstrdup(valptr);
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4555,6 +4557,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2fecf26a2c..6c3d2190b7 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -338,6 +338,9 @@ pgstat_bestart(void)
lbeentry.st_xact_start_timestamp = 0;
lbeentry.st_databaseid = MyDatabaseId;
+ lbeentry.st_tx_raw_bytes = lbeentry.st_rx_raw_bytes =
+ lbeentry.st_tx_compressed_bytes = lbeentry.st_rx_compressed_bytes = 0;
+
/* We have userid for client-backends, wal-sender and bgworker processes */
if (lbeentry.st_backendType == B_BACKEND
|| lbeentry.st_backendType == B_WAL_SENDER
@@ -716,6 +719,33 @@ pgstat_report_xact_timestamp(TimestampTz tstamp)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Report current transaction start timestamp as the specified value.
+ * Zero means there is no active transaction.
+ */
+void
+pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ if (!pgstat_track_activities || !beentry)
+ return;
+
+ /*
+ * Update my status entry, following the protocol of bumping
+ * st_changecount before and after. We use a volatile pointer here to
+ * ensure the compiler doesn't try to get cute.
+ */
+ PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
+
+ beentry->st_rx_raw_bytes += rx_raw_bytes;
+ beentry->st_tx_raw_bytes += tx_raw_bytes;
+ beentry->st_rx_compressed_bytes += rx_compressed_bytes;
+ beentry->st_tx_compressed_bytes += tx_compressed_bytes;
+
+ PGSTAT_END_WRITE_ACTIVITY(beentry);
+}
+
/* ----------
* pgstat_read_current_status() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..cff5dbf420 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -567,7 +567,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
Datum
pg_stat_get_activity(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_ACTIVITY_COLS 30
+#define PG_STAT_GET_ACTIVITY_COLS 34
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
int pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
@@ -916,6 +916,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[29] = true;
else
values[29] = UInt64GetDatum(beentry->st_query_id);
+ values[30] = beentry->st_rx_raw_bytes;
+ values[31] = beentry->st_tx_raw_bytes;
+ values[32] = beentry->st_rx_compressed_bytes;
+ values[33] = beentry->st_tx_compressed_bytes;
}
else
{
@@ -944,6 +948,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[27] = true;
nulls[28] = true;
nulls[29] = true;
+ nulls[30] = true;
+ nulls[31] = true;
+ nulls[32] = true;
+ nulls[33] = true;
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -1143,6 +1151,46 @@ pg_stat_get_backend_start(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(result);
}
+Datum
+pg_stat_get_network_traffic(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_NETWORK_TRAFFIC_COLS 4
+ TupleDesc tupdesc;
+ Datum values[PG_STAT_NETWORK_TRAFFIC_COLS];
+ bool nulls[PG_STAT_NETWORK_TRAFFIC_COLS];
+ int32 beid = PG_GETARG_INT32(0);
+ PgBackendStatus *beentry;
+
+ if ((beentry = pgstat_fetch_stat_beentry(beid)) == NULL)
+ PG_RETURN_NULL();
+ else if (!HAS_PGSTAT_PERMISSIONS(beentry->st_userid))
+ PG_RETURN_NULL();
+
+ /* Initialise values and NULL flags arrays */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(PG_STAT_NETWORK_TRAFFIC_COLS);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "rx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "tx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "rx_compressed_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "tx_compressed_bytes",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ /* Fill values and NULLs */
+ values[0] = beentry->st_rx_raw_bytes;
+ values[1] = beentry->st_tx_raw_bytes;
+ values[2] = beentry->st_rx_compressed_bytes;
+ values[3] = beentry->st_tx_compressed_bytes;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
Datum
pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6fc5cbc09a..fdf77e383d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -51,6 +51,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "common/string.h"
+#include "common/zpq_stream.h"
#include "funcapi.h"
#include "jit/jit.h"
#include "libpq/auth.h"
@@ -191,6 +192,7 @@ static bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
static bool check_log_stats(bool *newval, void **extra, GucSource source);
static bool check_canonical_path(char **newval, void **extra, GucSource source);
static bool check_timezone_abbreviations(char **newval, void **extra, GucSource source);
+static bool check_libpq_compression(char **newval, void **extra, GucSource source);
static void assign_timezone_abbreviations(const char *newval, void *extra);
static void pg_timezone_abbrev_initialize(void);
static const char *show_archive_command(void);
@@ -4581,6 +4583,16 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"libpq_compression", PGC_SIGHUP, CLIENT_CONN_OTHER,
+ gettext_noop("Sets the list of allowed libpq compression algorithms."),
+ NULL
+ },
+ &libpq_compress_algorithms,
+ "off",
+ check_libpq_compression, NULL, NULL
+ },
+
{
{"application_name", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Sets the application name to be reported in statistics and logs."),
@@ -5013,6 +5025,8 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
@@ -11845,6 +11859,22 @@ check_ssl(bool *newval, void **extra, GucSource source)
return true;
}
+static bool
+check_libpq_compression(char **newval, void **extra, GucSource source)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(*newval, &compressors, &n_compressors))
+ {
+ GUC_check_errdetail("Cannot parse the libpq_compression setting.");
+ return false;
+ }
+
+ free(compressors);
+ return true;
+}
+
static bool
check_stage_log_stats(bool *newval, void **extra, GucSource source)
{
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97f2a1f80a..554835e08e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6661,6 +6661,9 @@ threadRun(void *arg)
int nsocks; /* number of sockets to be waited for */
pg_time_usec_t min_usec;
pg_time_usec_t now = 0; /* set this only if needed */
+ bool buffered_rx = false; /* true if some of the clients has
+ * data left in SSL/ZPQ read
+ * buffers */
/*
* identify which client sockets should be checked for input, and
@@ -6695,6 +6698,9 @@ threadRun(void *arg)
*/
int sock = PQsocket(st->con);
+ /* check if conn has buffered SSL / ZPQ read data */
+ buffered_rx = buffered_rx || PQreadPending(st->con);
+
if (sock < 0)
{
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
@@ -6738,7 +6744,7 @@ threadRun(void *arg)
{
if (nsocks > 0)
{
- rc = wait_on_socket_set(sockets, min_usec);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, min_usec);
}
else /* nothing active, simple sleep */
{
@@ -6747,7 +6753,7 @@ threadRun(void *arg)
}
else /* no explicit delay, wait without timeout */
{
- rc = wait_on_socket_set(sockets, 0);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, 0);
}
if (rc < 0)
@@ -6786,8 +6792,11 @@ threadRun(void *arg)
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
goto done;
}
-
- if (!socket_has_input(sockets, sock, nsocks++))
+ if (PQreadPending(st->con))
+ {
+ nsocks++;
+ }
+ else if (!socket_has_input(sockets, sock, nsocks++))
continue;
}
else if (st->state == CSTATE_FINISHED ||
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index f590474855..f28c77aafc 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -171,6 +171,7 @@ static void print_with_linenumbers(FILE *output, char *lines,
const char *header_keyword);
static void minimal_error_message(PGresult *res);
+static void printCompressionInfo(void);
static void printSSLInfo(void);
static void printGSSInfo(void);
static bool printPsetInfo(const char *param, printQueryOpt *popt);
@@ -640,6 +641,7 @@ exec_command_conninfo(PsqlScanState scan_state, bool active_branch)
printf(_("You are connected to database \"%s\" as user \"%s\" on host \"%s\" at port \"%s\".\n"),
db, PQuser(pset.db), host, PQport(pset.db));
}
+ printCompressionInfo();
printSSLInfo();
printGSSInfo();
}
@@ -3656,6 +3658,21 @@ connection_warnings(bool in_startup)
}
}
+/*
+ * printCompressionInfo
+ *
+ * Print information about used compressor/decompressor
+ */
+static void
+printCompressionInfo(void)
+{
+ char *compressor = PQcompressor(pset.db);
+ char *decompressor = PQdecompressor(pset.db);
+
+ printf(_("Compression: client: %s, server: %s\n"),
+ compressor ? compressor : "none",
+ decompressor ? decompressor : "none");
+}
/*
* printSSLInfo
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..185c96063f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,7 +78,9 @@ OBJS_COMMON = \
unicode_norm.o \
username.o \
wait_error.o \
- wchar.o
+ wchar.o \
+ z_stream.o \
+ zpq_stream.o
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
new file mode 100644
index 0000000000..d04f22b924
--- /dev/null
+++ b/src/common/z_stream.c
@@ -0,0 +1,655 @@
+#include "c.h"
+#include "pg_config.h"
+#include "common/z_stream.h"
+
+/*
+ * Functions implementing streaming compression algorithm
+ */
+typedef struct
+{
+ /*
+ * Name of compression algorithm.
+ */
+ char const *(*name) (void);
+
+ /*
+ * Create new compression stream. level: compression level
+ */
+ void *(*create_compressor) (int level);
+
+ /*
+ * Create new decompression stream.
+ */
+ void *(*create_decompressor) ();
+
+ /*
+ * Decompress up to "src_size" compressed bytes from *src and write up to
+ * "dst_size" raw (decompressed) bytes to *dst. Number of decompressed
+ * bytes written to *dst is stored in *dst_processed. Number of compressed
+ * bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during decompression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * decompressor internal buffers.
+ *
+ * ZS_STREAM_END if encountered end of compressed data stream.
+ *
+ * ZS_DECOMPRESS_ERROR if encountered an error during decompression
+ * attempt.
+ */
+ ssize_t (*decompress) (void *ds, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Compress up to "src_size" raw (non-compressed) bytes from *src and
+ * write up to "dst_size" compressed bytes to *dst. Number of compressed
+ * bytes written to *dst is stored in *dst_processed. Number of
+ * non-compressed bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during compression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * compressor internal buffers.
+ *
+ * ZS_COMPRESS_ERROR if encountered an error during compression attempt.
+ */
+ ssize_t (*compress) (void *cs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Free compression stream created by create_compressor function.
+ */
+ void (*free_compressor) (void *cs);
+
+ /*
+ * Free decompression stream created by create_decompressor function.
+ */
+ void (*free_decompressor) (void *ds);
+
+ /*
+ * Get compressor error message.
+ */
+ char const *(*compress_error) (void *cs);
+
+ /*
+ * Get decompressor error message.
+ */
+ char const *(*decompress_error) (void *ds);
+
+ ssize_t (*end_compression) (void *cs, void *dst, size_t dst_size, size_t *dst_processed);
+} ZAlgorithm;
+
+struct ZStream
+{
+ ZAlgorithm const *algorithm;
+ void *stream;
+ bool not_flushed;
+};
+
+#if HAVE_LIBZ
+
+#include <stdlib.h>
+#include <zlib.h>
+
+
+static void *
+zlib_create_compressor(int level)
+{
+ int rc;
+ z_stream *c_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(c_stream, 0, sizeof(*c_stream));
+ rc = deflateInit(c_stream, level);
+ if (rc != Z_OK)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+zlib_create_decompressor()
+{
+ int rc;
+ z_stream *d_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(d_stream, 0, sizeof(*d_stream));
+ rc = inflateInit(d_stream);
+ if (rc != Z_OK)
+ {
+ free(d_stream);
+ return NULL;
+ }
+ return d_stream;
+}
+
+static ssize_t
+zlib_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *ds = (z_stream *) d_stream;
+ int rc;
+
+ ds->next_in = (Bytef *) src;
+ ds->avail_in = src_size;
+ ds->next_out = (Bytef *) dst;
+ ds->avail_out = dst_size;
+
+ rc = inflate(ds, Z_SYNC_FLUSH);
+ *src_processed = src_size - ds->avail_in;
+ *dst_processed = dst_size - ds->avail_out;
+
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_STREAM_END;
+ }
+ if (rc != Z_OK && rc != Z_BUF_ERROR)
+ {
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc PG_USED_FOR_ASSERTS_ONLY;
+ unsigned deflate_pending = 0;
+
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = (Bytef *) src;
+ cs->avail_in = src_size;
+
+ rc = deflate(cs, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ *dst_processed = dst_size - cs->avail_out;
+ *src_processed = src_size - cs->avail_in;
+
+ deflatePending(cs, &deflate_pending, Z_NULL); /* check if any data left
+ * in deflate buffer */
+ if (deflate_pending > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+zlib_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = NULL;
+ cs->avail_in = 0;
+
+ rc = deflate(cs, Z_STREAM_END);
+ Assert(rc == Z_OK || rc == Z_STREAM_END);
+ *dst_processed = dst_size - cs->avail_out;
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_OK;
+ }
+
+ return ZS_DATA_PENDING;
+}
+
+static void
+zlib_free_compressor(void *c_stream)
+{
+ z_stream *cs = (z_stream *) c_stream;
+
+ if (cs != NULL)
+ {
+ deflateEnd(cs);
+ free(cs);
+ }
+}
+
+static void
+zlib_free_decompressor(void *d_stream)
+{
+ z_stream *ds = (z_stream *) d_stream;
+
+ if (ds != NULL)
+ {
+ inflateEnd(ds);
+ free(ds);
+ }
+}
+
+static char const *
+zlib_error(void *stream)
+{
+ z_stream *zs = (z_stream *) stream;
+
+ return zs->msg;
+}
+
+static char const *
+zlib_name(void)
+{
+ return "zlib";
+}
+
+#endif
+
+#if USE_LZ4
+#include <lz4.h>
+
+#define MESSAGE_MAX_BYTES 819200
+#define RING_BUFFER_BYTES (1024 * 64 + MESSAGE_MAX_BYTES)
+
+typedef struct ZS_LZ4_CStream
+{
+ LZ4_stream_t *stream;
+ int level;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_CStream;
+
+typedef struct ZS_LZ4_DStream
+{
+ LZ4_streamDecode_t *stream;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_DStream;
+
+static void *
+lz4_create_compressor(int level)
+{
+ ZS_LZ4_CStream *c_stream = (ZS_LZ4_CStream *) malloc(sizeof(ZS_LZ4_CStream));
+ if (c_stream == NULL) {
+ return NULL;
+ }
+ c_stream->stream = LZ4_createStream();
+ c_stream->level = level;
+ c_stream->buf_pos = 0;
+ if (c_stream->stream == NULL)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+lz4_create_decompressor()
+{
+ ZS_LZ4_DStream *d_stream = (ZS_LZ4_DStream *) malloc(sizeof(ZS_LZ4_DStream));
+ if (d_stream == NULL) {
+ return NULL;
+ }
+
+ d_stream->stream = LZ4_createStreamDecode();
+ d_stream->buf_pos = 0;
+ if (d_stream->stream == NULL)
+ {
+ free(d_stream);
+ return NULL;
+ }
+
+ return d_stream;
+}
+
+static ssize_t
+lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+ char *const decPtr = &ds->buf[ds->buf_pos];
+ const int decBytes = LZ4_decompress_safe_continue(
+ ds->stream, src, decPtr, (int) src_size, (int) dst_size);
+
+ Assert(decBytes > 0);
+
+ *dst_processed = decBytes;
+ *src_processed = src_size;
+
+ memcpy(dst, decPtr, decBytes); /* write msg length */
+
+ ds->buf_pos += decBytes;
+ if (ds->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ ds->buf_pos = 0;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+ int cmpBytes;
+
+ src_size = Min(MESSAGE_MAX_BYTES, src_size);
+
+ memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
+
+ Assert(dst_size >= LZ4_compressBound(src_size));
+
+ cmpBytes = LZ4_compress_fast_continue(
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, cs->level);
+
+ Assert(cmpBytes > 0);
+ Assert(cmpBytes <= MESSAGE_MAX_BYTES);
+
+ *dst_processed = cmpBytes;
+ *src_processed = src_size;
+
+ cs->buf_pos += src_size;
+ if (cs->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ cs->buf_pos = 0;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+lz4_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ *dst_processed = 0;
+ return ZS_OK;
+}
+
+static void
+lz4_free_compressor(void *c_stream)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ if (cs->stream != NULL) {
+ LZ4_freeStream(cs->stream);
+ }
+ free(cs);
+ }
+}
+
+static void
+lz4_free_decompressor(void *d_stream)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ if (ds->stream != NULL) {
+ LZ4_freeStreamDecode(ds->stream);
+ }
+ free(ds);
+ }
+}
+
+static char const *
+lz4_error(void *stream)
+{
+ // lz4 doesn't have any explicit API to get the error names
+ return "NO_MSG";
+}
+
+static char const *
+lz4_name(void)
+{
+ return "lz4";
+}
+
+#endif
+
+static char const *
+no_compression_name(void)
+{
+ return NULL;
+}
+
+/*
+ * Array with all supported compression algorithms.
+ */
+static ZAlgorithm const zs_algorithms[] =
+{
+#if HAVE_LIBZ
+ {zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
+#endif
+#if USE_LZ4
+ {lz4_name, lz4_create_compressor, lz4_create_decompressor, lz4_decompress, lz4_compress, lz4_free_compressor, lz4_free_decompressor, lz4_error, lz4_error, lz4_end},
+#endif
+ {no_compression_name}
+};
+
+inline bool
+zs_is_valid_impl_id(unsigned int id)
+{
+ return id >= 0 && id < (sizeof(zs_algorithms) / sizeof(*zs_algorithms));
+}
+
+static ssize_t
+zs_init_compressor(ZStream * zs, unsigned int c_alg_impl, int c_level)
+{
+ if (!zs_is_valid_impl_id(c_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[c_alg_impl];
+ zs->stream = zs->algorithm->create_compressor(c_level);
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+static ssize_t
+zs_init_decompressor(ZStream * zs, unsigned int d_alg_impl)
+{
+ if (!zs_is_valid_impl_id(d_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[d_alg_impl];
+ zs->stream = zs->algorithm->create_decompressor();
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Index of used compression algorithm in zs_algorithms array.
+ */
+ZStream *
+zs_create_compressor(unsigned int c_alg_impl, int c_level)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_compressor(zs, c_alg_impl, c_level))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ZStream *
+zs_create_decompressor(unsigned int d_alg_impl)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_decompressor(zs, d_alg_impl))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ssize_t
+zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *src_processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->decompress(zs->stream,
+ src, src_size, src_processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+
+ if (rc == ZS_OK || rc == ZS_INCOMPLETE_SRC || rc == ZS_STREAM_END) {
+ return rc;
+ }
+
+ return ZS_DECOMPRESS_ERROR;
+}
+
+ssize_t
+zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->compress(zs->stream,
+ buf, size, processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+void
+zs_compressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_compressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+void
+zs_decompressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_decompressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+ssize_t
+zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *dst_processed = 0;
+
+ rc = zs->algorithm->end_compression(zs->stream, dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+char const *
+zs_compress_error(ZStream * zs)
+{
+ return zs->algorithm->compress_error(zs->stream);
+}
+
+char const *
+zs_decompress_error(ZStream * zs)
+{
+ return zs->algorithm->decompress_error(zs->stream);
+}
+
+bool
+zs_buffered(ZStream * zs)
+{
+ return zs ? zs->not_flushed : 0;
+}
+
+
+/*
+ * Get list of the supported algorithms.
+ */
+char **
+zs_get_supported_algorithms(void)
+{
+ size_t n_algorithms = sizeof(zs_algorithms) / sizeof(*zs_algorithms);
+ char **algorithm_names = malloc(n_algorithms * sizeof(char *));
+
+ for (size_t i = 0; i < n_algorithms; i++)
+ {
+ algorithm_names[i] = (char *) zs_algorithms[i].name();
+ }
+
+ return algorithm_names;
+}
+
+char const *
+zs_compress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
+
+char const *
+zs_decompress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000000..c72b434c5d
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,1007 @@
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+#include <unistd.h>
+#include <math.h>
+
+#include "common/zpq_stream.h"
+#include "pg_config.h"
+#include "port/pg_bswap.h"
+
+/* log warnings on backend */
+#ifndef FRONTEND
+#define pg_log_warning(...) elog(WARNING, __VA_ARGS__)
+#else
+#define pg_log_warning(...) (void)0
+#endif
+
+/* ZpqBuffer size, in bytes */
+#define ZPQ_BUFFER_SIZE 8192000
+/* CompressedData msg_type */
+#define ZPQ_COMPRESSED_MSG_TYPE 'm'
+/* SetCompressionMethod msg_type */
+#define ZPQ_SET_COMPRESSION_MSG_TYPE 'k'
+
+#define ZPQ_COMPRESS_THRESHOLD 60
+
+typedef struct ZpqBuffer ZpqBuffer;
+
+
+/* ZpqBuffer used as RX/TX buffer in ZpqStream */
+struct ZpqBuffer
+{
+ char buf[ZPQ_BUFFER_SIZE];
+ size_t size; /* current size of buf */
+ size_t pos; /* current position in buf, in range [0, size] */
+};
+
+static inline void
+zpq_buf_init(ZpqBuffer * zb)
+{
+ zb->size = 0;
+ zb->pos = 0;
+}
+
+static inline size_t
+zpq_buf_left(ZpqBuffer * zb)
+{
+ Assert(zb->buf);
+ return ZPQ_BUFFER_SIZE - zb->size;
+}
+
+static inline size_t
+zpq_buf_unread(ZpqBuffer * zb)
+{
+ return zb->size - zb->pos;
+}
+
+static inline char *
+zpq_buf_size(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->size;
+}
+
+static inline char *
+zpq_buf_pos(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->pos;
+}
+
+static inline void
+zpq_buf_size_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->size += value;
+}
+
+static inline void
+zpq_buf_pos_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->pos += value;
+}
+
+static inline void
+zpq_buf_reuse(ZpqBuffer * zb)
+{
+ size_t unread = zpq_buf_unread(zb);
+
+ if (unread > 5) /* can read message header, don't do anything */
+ return;
+ if (unread == 0)
+ {
+ zb->size = 0;
+ zb->pos = 0;
+ return;
+ }
+ memmove(zb->buf, zb->buf + zb->pos, unread);
+ zb->size = unread;
+ zb->pos = 0;
+}
+
+struct ZpqStream
+{
+ ZStream *c_stream; /* underlying compression stream */
+ ZStream *d_stream; /* underlying decompression stream */
+
+ size_t tx_total; /* amount of bytes sent to tx_func */
+
+ size_t tx_total_raw; /* amount of bytes received by zpq_write */
+ size_t rx_total; /* amount of bytes read by rx_func */
+ size_t rx_total_raw; /* amount of bytes returned by zpq_write */
+ bool is_compressing; /* current compression state */
+
+ bool is_decompressing; /* current decompression state */
+ size_t rx_msg_bytes_left; /* number of bytes left to process without
+ * changing the decompression state */
+ size_t tx_msg_bytes_left; /* number of bytes left to process without
+ * changing the compression state */
+
+ ZpqBuffer rx_in; /* buffer for unprocessed data read by rx_func */
+ ZpqBuffer tx_in; /* buffer for unprocessed data consumed by
+ * zpq_write */
+ ZpqBuffer tx_out; /* buffer for processed data waiting for send
+ * via tx_func */
+
+ zpq_rx_func rx_func;
+ zpq_tx_func tx_func;
+ void *arg;
+
+ zpq_compressor *compressors; /* compressors array holds the available
+ * compressors to use for
+ * compression/decompression */
+ size_t n_compressors; /* size of the compressors array */
+ int compress_alg_idx; /* index of the active compression
+ * algorithm */
+ int decompress_alg_idx; /* index of the active decompression
+ * algorithm */
+ int compressor_by_msg_type[256]; /* map to choose a compressor
+ * by the protocol message
+ * type */
+
+ bool reading_set_compression; /* utility marker indicating
+ * partial SetCompressionMethod
+ * read */
+};
+
+/*
+ * Message compression map defines the logic for choosing the compressor
+ * based on the protocol message type. Currently, it is a basic prototype to demonstrate
+ * the capabilities of the on-the-fly compression switch.
+ */
+static inline void
+zpq_build_msg_compression_map(ZpqStream * zpq)
+{
+ int i;
+
+ for (i = 0; i < 256; i++)
+ {
+ zpq->compressor_by_msg_type[i] = -1;
+ }
+
+ for (i = 0; i < zpq->n_compressors; i++)
+ {
+ /* compress CopyData, DataRow and Query messages */
+ if (zpq->compressor_by_msg_type['d'] == -1) {
+ zpq->compressor_by_msg_type['d'] = i;
+ }
+ if (zpq->compressor_by_msg_type['D'] == -1) {
+ zpq->compressor_by_msg_type['D'] = i;
+ }
+ if (zpq->compressor_by_msg_type['Q'] == -1) {
+ zpq->compressor_by_msg_type['Q'] = i;
+ }
+ }
+}
+
+/*
+ * Choose the index of compressor to use for the message of msg_type with msg_len.
+ * Return values:
+ * - the non-negative index of zpq->compressors array
+ * - -1, if message should not be compressed
+ */
+static inline int
+zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
+ {
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
+ }
+ return -1;
+}
+
+/*
+ * Check if should compress message of msg_type with msg_len.
+ * Return true if should, false if should not.
+ */
+static inline bool
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
+}
+
+/*
+ * Check if message is a CompressedData.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_compressed_msg(char msg_type)
+{
+ return msg_type == ZPQ_COMPRESSED_MSG_TYPE;
+}
+
+/*
+ * Check if message is a SetCompressionMethod.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_set_compression_msg(char msg_type)
+{
+ return msg_type == ZPQ_SET_COMPRESSION_MSG_TYPE;
+}
+
+ZpqStream *
+zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size)
+{
+ ZpqStream *zpq;
+
+ /* zpqStream needs at least one compressor */
+ if (n_compressors == 0 || compressors == NULL)
+ {
+ return NULL;
+ }
+ zpq = (ZpqStream *) malloc(sizeof(ZpqStream));
+
+ zpq->compressors = compressors;
+ zpq->n_compressors = n_compressors;
+ zpq->compress_alg_idx = -1;
+ zpq->decompress_alg_idx = -1;
+
+ zpq->is_compressing = false;
+ zpq->is_decompressing = false;
+ zpq->rx_msg_bytes_left = 0;
+ zpq->tx_msg_bytes_left = 0;
+ zpq_buf_init(&zpq->tx_in);
+
+ zpq->tx_total = 0;
+ zpq->tx_total_raw = 0;
+ zpq->rx_total = 0;
+ zpq->rx_total_raw = 0;
+
+ zpq_buf_init(&zpq->rx_in);
+ zpq_buf_size_advance(&zpq->rx_in, rx_data_size);
+ Assert(rx_data_size < ZPQ_BUFFER_SIZE);
+ memcpy(zpq->rx_in.buf, rx_data, rx_data_size);
+
+ zpq_buf_init(&zpq->tx_out);
+
+ zpq->rx_func = rx_func;
+ zpq->tx_func = tx_func;
+ zpq->arg = arg;
+ zpq->reading_set_compression = false;
+ zpq->c_stream = NULL;
+ zpq->d_stream = NULL;
+
+ zpq_build_msg_compression_map(zpq);
+
+ return zpq;
+}
+
+/* Compress up to src_size bytes from *src into CompressedData and write it to the tx buffer.
+ * Returns ZS_OK on success, ZS_COMPRESS_ERROR if encountered a compression error. */
+static inline ssize_t
+zpq_write_compressed_message(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ size_t compressed_len;
+ ssize_t rc;
+ uint32 size;
+
+ /* check if have enough space */
+ if (zpq_buf_left(&zpq->tx_out) <= 5)
+ {
+ /* too little space for CompressedData, abort */
+ *src_processed = 0;
+ return ZS_OK;
+ }
+
+ compressed_len = 0;
+ rc = zs_write(zpq->c_stream, src, src_size, src_processed,
+ zpq_buf_size(&zpq->tx_out) + 5, zpq_buf_left(&zpq->tx_out) - 5, &compressed_len);
+
+ if (compressed_len > 0)
+ {
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_COMPRESSED_MSG_TYPE;
+ size = pg_hton32(compressed_len + 4);
+
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &size, sizeof(uint32)); /* write msg length */
+ compressed_len += 5; /* append header length to compressed data
+ * length */
+ }
+
+ zpq->tx_total_raw += *src_processed;
+ zpq->tx_total += compressed_len;
+ zpq_buf_size_advance(&zpq->tx_out, compressed_len);
+ return rc;
+}
+
+/* Copy the data directly from *src to the tx buffer */
+static void
+zpq_write_uncompressed(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ src_size = Min(zpq_buf_left(&zpq->tx_out), src_size);
+ memcpy(zpq_buf_size(&zpq->tx_out), src, src_size);
+
+ zpq->tx_total_raw += src_size;
+ zpq->tx_total += src_size;
+ zpq_buf_size_advance(&zpq->tx_out, src_size);
+ *src_processed = src_size;
+}
+
+static ssize_t
+zpq_write_set_compression_msg(ZpqStream * zpq, int new_compress_idx)
+{
+ uint32 len;
+ uint8 idx;
+
+ /*
+ * check if have enough space: msg_type(1 byte) + msg_len(4 bytes) +
+ * compress_alg_idx(1 byte)
+ */
+ if (zpq_buf_left(&zpq->tx_out) < 6)
+ {
+ return -1;
+ }
+
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_SET_COMPRESSION_MSG_TYPE;
+ len = pg_hton32(5);
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &len, sizeof(uint32)); /* write msg length */
+
+ /* currently we expect idx to be in range [0, 255] */
+ Assert(new_compress_idx >= 0 && new_compress_idx <= UINT8_MAX);
+ idx = (uint8) new_compress_idx;
+ memcpy(zpq_buf_size(&zpq->tx_out) + 5, &idx, sizeof(uint8)); /* write
+ * new_compress_idx */
+
+ zpq->tx_total_raw += 6;
+ zpq->tx_total += 6;
+ zpq_buf_size_advance(&zpq->tx_out, 6);
+ return 0;
+}
+
+/* Determine if should compress the next message and change the current compression state */
+static ssize_t
+zpq_toggle_compression(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ int new_compress_idx = zpq_choose_compressor(zpq, msg_type, msg_len);
+ bool should_compress = new_compress_idx != -1;
+
+ /*
+ * negative new_compress_idx indicates that we should not compress this
+ * message
+ */
+ if (should_compress)
+ {
+ /*
+ * if the new compressor does not match the current one, process the
+ * switch
+ */
+ if (zpq->compress_alg_idx != new_compress_idx)
+ {
+ if (zpq_write_set_compression_msg(zpq, new_compress_idx))
+ {
+ /*
+ * come back later when we can write the entire
+ * SetCompressionMethod message
+ */
+ return 0;
+ }
+
+ zs_compressor_free(zpq->c_stream);
+ zpq->c_stream = zs_create_compressor(zpq->compressors[new_compress_idx].impl, zpq->compressors[new_compress_idx].level);
+ if (zpq->c_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->compress_alg_idx = new_compress_idx;
+ }
+ }
+
+ zpq->is_compressing = should_compress;
+ zpq->tx_msg_bytes_left = msg_len + 1;
+ return 0;
+}
+
+/*
+ * Internal write function. Reads the data from *src buffer,
+ * determines the postgres messages type and length.
+ * If message matches the compression criteria, it wraps the message into
+ * CompressedData. Otherwise, leaves the message unchanged.
+ * If *src data ends with incomplete message header, this function is not
+ * going to read this message header.
+ * Returns number of written raw bytes or error code.
+ * In the last case number of bytes written is stored in *processed.
+ */
+static ssize_t
+zpq_write_internal(ZpqStream * zpq, void const *src, size_t src_size, size_t *processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ do
+ {
+ /*
+ * try to read ahead the next message types and increase
+ * tx_msg_bytes_left, if possible
+ */
+ while (zpq->tx_msg_bytes_left > 0 && src_size - src_pos >= zpq->tx_msg_bytes_left + 5)
+ {
+ char msg_type = *((char *) src + src_pos + zpq->tx_msg_bytes_left);
+ uint32 msg_len;
+
+ memcpy(&msg_len, (char *) src + src_pos + zpq->tx_msg_bytes_left + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ if (zpq_should_compress(zpq, msg_type, msg_len) != zpq->is_compressing)
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+ zpq->tx_msg_bytes_left += msg_len + 1;
+ }
+
+ /*
+ * Write CompressedData if currently is compressing or have some
+ * buffered data left in underlying compression stream
+ */
+ if (zs_buffered(zpq->c_stream) || (zpq->is_compressing && zpq->tx_msg_bytes_left > 0))
+ {
+ size_t buf_processed = 0;
+ size_t to_compress = Min(zpq->tx_msg_bytes_left, src_size - src_pos);
+
+ rc = zpq_write_compressed_message(zpq, (char *) src + src_pos, to_compress, &buf_processed);
+ src_pos += buf_processed;
+ zpq->tx_msg_bytes_left -= buf_processed;
+
+ if (rc != ZS_OK)
+ {
+ *processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * If not going to compress the data from *src, just write it
+ * uncompressed.
+ */
+ else if (zpq->tx_msg_bytes_left > 0)
+ { /* determine next message type */
+ size_t copy_len = Min(src_size - src_pos, zpq->tx_msg_bytes_left);
+ size_t copy_processed = 0;
+
+ zpq_write_uncompressed(zpq, (char *) src + src_pos, copy_len, ©_processed);
+ src_pos += copy_processed;
+ zpq->tx_msg_bytes_left -= copy_processed;
+ }
+
+ /*
+ * Reached the compression toggle point, fetch next message header to
+ * determine compression state.
+ */
+ else
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ if (src_size - src_pos < 5)
+ {
+ /*
+ * must return here because we can't continue without full
+ * message header
+ */
+ *processed = src_pos;
+ return ZPQ_INCOMPLETE_HEADER;
+ }
+
+ msg_type = *((char *) src + src_pos);
+ memcpy(&msg_len, (char *) src + src_pos + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ rc = zpq_toggle_compression(zpq, msg_type, msg_len);
+ if (rc)
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * repeat sending while there is some data in input or internal
+ * compression buffer
+ */
+ } while (src_pos < src_size && zpq_buf_left(&zpq->tx_out) > 6);
+
+ return src_pos;
+}
+
+ssize_t
+zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ /* try to process as much data as possible before calling the tx_func */
+ while (zpq_buf_left(&zpq->tx_out) > 6)
+ {
+ size_t copy_len = Min(zpq_buf_left(&zpq->tx_in), src_size - src_pos);
+ size_t processed;
+
+ memcpy(zpq_buf_size(&zpq->tx_in), (char *) src + src_pos, copy_len);
+ zpq_buf_size_advance(&zpq->tx_in, copy_len);
+ src_pos += copy_len;
+
+ if (zpq_buf_unread(&zpq->tx_in) == 0 && !zs_buffered(zpq->c_stream))
+ {
+ break;
+ }
+
+ processed = 0;
+
+ rc = zpq_write_internal(zpq, zpq_buf_pos(&zpq->tx_in), zpq_buf_unread(&zpq->tx_in), &processed);
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, rc);
+ zpq_buf_reuse(&zpq->tx_in);
+ }
+ else
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, processed);
+ zpq_buf_reuse(&zpq->tx_in);
+ if (rc == ZPQ_INCOMPLETE_HEADER)
+ {
+ break;
+ }
+ *src_processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * call the tx_func if have any bytes to send
+ */
+ while (zpq_buf_unread(&zpq->tx_out))
+ {
+ rc = zpq->tx_func(zpq->arg, zpq_buf_pos(&zpq->tx_out), zpq_buf_unread(&zpq->tx_out));
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_out, rc);
+ }
+ else
+ {
+ *src_processed = src_pos;
+ zpq_buf_reuse(&zpq->tx_out);
+ return rc;
+ }
+ }
+
+ zpq_buf_reuse(&zpq->tx_out);
+ return src_pos;
+}
+
+/* Decompress bytes from RX buffer and write up to dst_len of uncompressed data to *dst.
+ * Returns:
+ * ZS_OK on success,
+ * ZS_STREAM_END if reached end of compressed chunk
+ * ZS_DECOMPRESS_ERROR if encountered a decompression error */
+static inline ssize_t
+zpq_read_compressed_message(ZpqStream * zpq, char *dst, size_t dst_len, size_t *dst_processed)
+{
+ size_t rx_processed = 0;
+ ssize_t rc;
+ size_t read_len = Min(zpq->rx_msg_bytes_left, zpq_buf_unread(&zpq->rx_in));
+
+ Assert(read_len == zpq->rx_msg_bytes_left);
+ rc = zs_read(zpq->d_stream, zpq_buf_pos(&zpq->rx_in), read_len, &rx_processed,
+ dst, dst_len, dst_processed);
+
+ zpq_buf_pos_advance(&zpq->rx_in, rx_processed);
+ zpq->rx_total_raw += *dst_processed;
+ zpq->rx_msg_bytes_left -= rx_processed;
+ return rc;
+}
+
+/* Copy up to dst_len bytes from rx buffer to *dst.
+ * Returns amount of bytes copied. */
+static inline size_t
+zpq_read_uncompressed(ZpqStream * zpq, char *dst, size_t dst_len)
+{
+ size_t copy_len;
+
+ Assert(zpq_buf_unread(&zpq->rx_in) > 0);
+ copy_len = Min(zpq->rx_msg_bytes_left, Min(zpq_buf_unread(&zpq->rx_in), dst_len));
+
+ memcpy(dst, zpq_buf_pos(&zpq->rx_in), copy_len);
+
+ zpq_buf_pos_advance(&zpq->rx_in, copy_len);
+ zpq->rx_total_raw += copy_len;
+ zpq->rx_msg_bytes_left -= copy_len;
+ return copy_len;
+}
+
+/* Determine if should decompress the next message and
+ * change the current decompression state */
+static inline void
+zpq_toggle_decompression(ZpqStream * zpq)
+{
+ uint32 msg_len;
+ char msg_type = *zpq_buf_pos(&zpq->rx_in);
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+
+ if (zpq_is_set_compression_msg(msg_type))
+ {
+ Assert(msg_len == 5);
+ zpq->reading_set_compression = true;
+ /* set compression message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ }
+ else
+ {
+ zpq->is_decompressing = zpq_is_compressed_msg(msg_type);
+ zpq->rx_msg_bytes_left = msg_len + 1;
+
+ if (zpq->is_decompressing)
+ {
+ /* compressed message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ zpq->rx_msg_bytes_left -= 5;
+ }
+ }
+}
+
+static inline ssize_t
+zpq_process_switch(ZpqStream * zpq)
+{
+ uint8 algorithm_idx;
+
+ if (zpq_buf_unread(&zpq->rx_in) < 1)
+ {
+ return 0;
+ }
+
+ algorithm_idx = *zpq_buf_pos(&zpq->rx_in);
+
+ zpq_buf_pos_advance(&zpq->rx_in, 1);
+ zpq->reading_set_compression = false;
+
+ if (algorithm_idx != zpq->decompress_alg_idx)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ zpq->d_stream = zs_create_decompressor(zpq->compressors[algorithm_idx].impl);
+ if (zpq->d_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->decompress_alg_idx = algorithm_idx;
+ }
+
+ return 0;
+}
+
+ssize_t
+zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock)
+{
+ size_t dst_pos = 0;
+ size_t dst_processed = 0;
+ ssize_t rc;
+
+ /* Read until some data fetched */
+ while (dst_pos == 0)
+ {
+ zpq_buf_reuse(&zpq->rx_in);
+
+ if (!zpq_buffered_rx(zpq) || (zpq->is_decompressing && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left))
+ {
+ if (noblock) {
+ /* can't read anything w/o the potentially
+ * blocking backend call */
+ return dst_pos;
+ }
+ rc = zpq->rx_func(zpq->arg, zpq_buf_size(&zpq->rx_in), zpq_buf_left(&zpq->rx_in));
+ if (rc > 0) /* read fetches some data */
+ {
+ zpq->rx_total += rc;
+ zpq_buf_size_advance(&zpq->rx_in, rc);
+ }
+ else /* read failed */
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * try to read ahead the next message types and increase
+ * rx_msg_bytes_left, if possible (ONLY UNCOMPRESSED MESSAGES)
+ */
+ while (!zpq->is_decompressing && zpq->rx_msg_bytes_left > 0 && (zpq_buf_unread(&zpq->rx_in) >= zpq->rx_msg_bytes_left + 5))
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ msg_type = *(zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left);
+ if (zpq_is_compressed_msg(msg_type) || zpq_is_set_compression_msg(msg_type))
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left + 1, 4);
+ zpq->rx_msg_bytes_left += pg_ntoh32(msg_len) + 1;
+ }
+
+
+ if (zpq->rx_msg_bytes_left > 0 || zs_buffered(zpq->d_stream))
+ {
+ dst_processed = 0;
+ if (zpq->is_decompressing || zs_buffered(zpq->d_stream))
+ {
+ if (!zs_buffered(zpq->d_stream) && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left) {
+ /* prefer to read only the fully compressed messages or read if some data is buffered */
+ continue;
+ }
+ rc = zpq_read_compressed_message(zpq, dst, dst_size - dst_pos, &dst_processed);
+ dst_pos += dst_processed;
+ if (rc == ZS_STREAM_END)
+ {
+ continue;
+ }
+ if (rc != ZS_OK)
+ {
+ return rc;
+ }
+ }
+ else
+ dst_pos += zpq_read_uncompressed(zpq, dst, dst_size - dst_pos);
+ }
+ else if (zpq->reading_set_compression)
+ {
+ zpq_process_switch(zpq);
+ }
+ else if (zpq_buf_unread(&zpq->rx_in) >= 5)
+ zpq_toggle_decompression(zpq);
+ }
+ return dst_pos;
+}
+
+bool
+zpq_buffered_rx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->rx_in) >= 5 || (zpq_buf_unread(&zpq->rx_in) > 0 && zpq->rx_msg_bytes_left > 0) ||
+ zs_buffered(zpq->d_stream) : 0;
+}
+
+bool
+zpq_buffered_tx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->tx_in) >= 5 || (zpq_buf_unread(&zpq->tx_in) > 0 && zpq->tx_msg_bytes_left > 0) || zpq_buf_unread(&zpq->tx_out) > 0 ||
+ zs_buffered(zpq->c_stream) : 0;
+}
+
+void
+zpq_free(ZpqStream * zpq)
+{
+ if (zpq)
+ {
+ if (zpq->c_stream)
+ {
+ zs_compressor_free(zpq->c_stream);
+ }
+ if (zpq->d_stream)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ }
+ free(zpq);
+ }
+}
+
+char const *
+zpq_compress_error(ZpqStream * zpq)
+{
+ return zs_compress_error(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_error(ZpqStream * zpq)
+{
+ return zs_decompress_error(zpq->d_stream);
+}
+
+char const *
+zpq_compress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_compress_algorithm_name(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_decompress_algorithm_name(zpq->d_stream);
+}
+
+bool
+zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int i;
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *protocol_extension = strchr(val, ';');
+
+ *compressors = NULL;
+ *n_compressors = 0;
+
+ /* No protocol extensions are currently supported */
+ if (protocol_extension)
+ *protocol_extension = '\0';
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ if (pg_strcasecmp(val, "true") == 0 ||
+ pg_strcasecmp(val, "yes") == 0 ||
+ pg_strcasecmp(val, "on") == 0 ||
+ pg_strcasecmp(val, "any") == 0 ||
+ pg_strcasecmp(val, "1") == 0)
+ {
+ /* return all available compressors */
+ *n_compressors = n_supported_algorithms;
+
+ if (n_supported_algorithms)
+ {
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+ for (i = 0; i < n_supported_algorithms; i++)
+ {
+ (*compressors)[i].impl = i;
+ (*compressors)[i].level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ }
+ }
+ return true;
+ }
+
+ if (*val == 0 ||
+ pg_strcasecmp(val, "false") == 0 ||
+ pg_strcasecmp(val, "no") == 0 ||
+ pg_strcasecmp(val, "off") == 0 ||
+ pg_strcasecmp(val, "0") == 0)
+ {
+ /* Compression is disabled */
+ return true;
+ }
+
+
+ return zpq_deserialize_compressors(val, compressors, n_compressors);
+}
+
+bool
+zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int selected_alg_mask = 0; /* bitmask of already selected
+ * algorithms to avoid duplicates in
+ * compressors */
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *c_string_dup = strdup(c_string); /* following parsing can
+ * modify the string */
+ char *p = c_string_dup;
+
+ *n_compressors = 0;
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+
+ while (*p != '\0')
+ {
+ char *sep = strchr(p, ',');
+ char *col;
+ int compression_level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ bool found;
+
+ if (sep != NULL)
+ *sep = '\0';
+
+ col = strchr(p, ':');
+ if (col != NULL)
+ {
+ *col = '\0';
+ if (sscanf(col + 1, "%d", &compression_level) != 1)
+ {
+ pg_log_warning("invalid compression level %s in compression option '%s'", col + 1, p);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+ }
+ found = false;
+ for (int i = 0; supported_algorithms[i] != NULL; i++)
+ {
+ if (pg_strcasecmp(p, supported_algorithms[i]) == 0)
+ {
+ if (selected_alg_mask & (1 << i))
+ {
+ /* duplicates are not allowed */
+ pg_log_warning("duplicate algorithm %s in compressors string %s", p, c_string);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+
+ (*compressors)[*n_compressors].impl = i;
+ (*compressors)[*n_compressors].level = compression_level;
+
+ selected_alg_mask |= 1 << i;
+ *n_compressors += 1;
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ pg_log_warning("algorithm %s is not supported", p);
+ }
+ if (sep)
+ p = sep + 1;
+ else
+ break;
+ }
+
+ if (*n_compressors == 0)
+ {
+ free(*compressors);
+ *compressors = NULL;
+ }
+ free(c_string_dup);
+ return true;
+}
+
+char *
+zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors)
+{
+ char *res;
+ char *p;
+ size_t i;
+ size_t total_len = 0;
+ char **supported_algorithms = zs_get_supported_algorithms();
+
+ if (n_compressors == 0)
+ {
+ return NULL;
+ }
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ size_t level_len;
+
+ if (!zs_is_valid_impl_id(compressors[i].impl))
+ {
+ pg_log_warning("algorithm impl_id %d is incorrect", compressors[i].impl);
+ return NULL;
+ }
+
+ /* determine the length of the compression level string */
+ level_len = compressors[i].level == 0 ? 1 : (int) floor(log10(abs(compressors[i].level))) + 1;
+ if (compressors[i].level < 0)
+ {
+ level_len += 1; /* add the leading "-" */
+ }
+
+ /*
+ * single entry looks like "alg_name:compression_level," so +2 is for
+ * ":" and "," symbols (or trailing null)
+ */
+ total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
+ }
+
+ res = p = malloc(total_len);
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
+ if (i < n_compressors - 1)
+ *p++ = ',';
+ }
+ return res;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d6bf1f3274..ed7d360c86 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5334,9 +5334,9 @@
proname => 'pg_stat_get_activity', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'int4',
- proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8}',
- proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id}',
+ proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
prosrc => 'pg_stat_get_activity' },
{ oid => '3318',
descr => 'statistics: information about progress of backends running maintenance command',
@@ -5644,6 +5644,14 @@
proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,wal_write,wal_sync,wal_write_time,wal_sync_time,stats_reset}',
prosrc => 'pg_stat_get_wal' },
+{ oid => '9598', descr => 'statistics: information about network traffic',
+ proname => 'pg_stat_get_network_traffic', proisstrict => 'f', provolatile => 's',
+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o}',
+ proargnames => '{_beid,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
+ prosrc => 'pg_stat_get_network_traffic' },
+
{ oid => '2306', descr => 'statistics: information about SLRU caches',
proname => 'pg_stat_get_slru', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
@@ -5785,6 +5793,10 @@
proname => 'pg_tablespace_location', provolatile => 's', prorettype => 'text',
proargtypes => 'oid', prosrc => 'pg_tablespace_location' },
+{ oid => '9257', descr => 'connection compression algorithm',
+ proname => 'pg_compression_algorithm', provolatile => 's', prorettype => 'text',
+ proargtypes => '', prosrc => 'pg_compression_algorithm' },
+
{ oid => '1946',
descr => 'convert bytea value into some ascii-only text string',
proname => 'encode', prorettype => 'text', proargtypes => 'bytea text',
diff --git a/src/include/common/z_stream.h b/src/include/common/z_stream.h
new file mode 100644
index 0000000000..8229035978
--- /dev/null
+++ b/src/include/common/z_stream.h
@@ -0,0 +1,107 @@
+/*
+ * z_stream.h
+ * Streaming compression
+ */
+
+
+#ifndef Z_STREAM_H
+#define Z_STREAM_H
+
+#include <stdlib.h>
+
+#define ZS_OK (0)
+#define ZS_IO_ERROR (-1)
+#define ZS_DECOMPRESS_ERROR (-2)
+#define ZS_COMPRESS_ERROR (-3)
+#define ZS_STREAM_END (-4)
+#define ZS_DATA_PENDING (-5)
+#define ZS_INCOMPLETE_SRC (-6) /* cannot decompress unless full src message is fetched */
+#define ZS_EXPAND_DST (-7) /* cannot compress: dst buffer is too small for given src size */
+
+struct ZStream;
+typedef struct ZStream ZStream;
+
+#endif
+
+/*
+ * Create compression stream for sending compressed data.
+ * c_alg_impl: index of chosen compression algorithm
+ * c_level: compression c_level
+ */
+extern ZStream * zs_create_compressor(unsigned int c_alg_impl, int c_level);
+
+/*
+ * Create decompression stream for reading compressed data.
+ * d_alg_impl: index of chosen decompression algorithm
+ */
+extern ZStream * zs_create_decompressor(unsigned int d_alg_impl);
+
+/*
+ * Read up to "size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZS_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Write up to "size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZS_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *processed.
+ */
+extern ssize_t zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zs_decompress_error(ZStream * zs);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zs_compress_error(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal rx decompression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal tx compression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * End the compression stream.
+ */
+extern ssize_t zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Free stream created by zs_create_compressor function.
+ */
+extern void zs_compressor_free(ZStream * zs);
+
+/*
+ * Free stream created by zs_create_decompressor function.
+ */
+extern void zs_decompressor_free(ZStream * zs);
+
+/*
+ * Get the name of chosen compression algorithm.
+ */
+extern char const *zs_compress_algorithm_name(ZStream * zs);
+
+/*
+ * Get the name of chosen decompression algorithm.
+ */
+extern char const *zs_decompress_algorithm_name(ZStream * zs);
+
+/*
+ Returns zero terminated array with compression algorithms names
+*/
+extern char **zs_get_supported_algorithms(void);
+
+/*
+ Returns true if provided id is a valid compression algorithm id, otherwise returns false
+*/
+extern bool zs_is_valid_impl_id(unsigned int id);
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000000..bf81429707
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,117 @@
+/*
+ * zpq_stream.h
+ * Streaming compression for libpq
+ */
+#include "z_stream.h"
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#define ZPQ_DEFAULT_COMPRESSION_LEVEL (1)
+#define ZPQ_INCOMPLETE_HEADER (-6)
+#define ZPQ_FATAL_ERROR (-7)
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t (*zpq_tx_func) (void *arg, void const *data, size_t size);
+typedef ssize_t (*zpq_rx_func) (void *arg, void *data, size_t size);
+
+/*
+ * Descriptor of compression algorithm chosen by client
+ */
+typedef struct zpq_compressor
+{
+ unsigned int impl; /* compression algorithm index */
+ int level; /* compression level */
+} zpq_compressor;
+
+#endif
+
+/*
+ * Create compression stream with rx/tx function for reading/sending compressed data.
+ * tx_func: function for writing compressed data in underlying stream
+ * rx_func: function for reading compressed data from underlying stream
+ * arg: context passed to the function
+ * rx_data: received data (compressed data already fetched from input stream)
+ * rx_data_size: size of data fetched from input stream
+ */
+extern ZpqStream * zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size);
+
+/*
+ * Write up to "src_size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZPQ_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *src_processed.
+ */
+extern ssize_t zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed);
+
+/*
+ * Read up to "dst_size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZPQ_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock);
+
+/*
+ * Return true if non-flushed data left in internal rx decompression buffer.
+ */
+extern bool zpq_buffered_rx(ZpqStream * zpq);
+
+/*
+ * Return true if non-flushed data left in internal tx compression buffer.
+ */
+extern bool zpq_buffered_tx(ZpqStream * zpq);
+
+/*
+ * Free stream created by zs_create function.
+ */
+extern void zpq_free(ZpqStream * zpq);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zpq_decompress_error(ZpqStream * zpq);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zpq_compress_error(ZpqStream * zpq);
+
+/*
+ * Get the name of the current compression algorithm.
+ */
+extern char const *zpq_compress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Get the name of the current decompression algorithm.
+ */
+extern char const *zpq_decompress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Parse comma-separated list of compression algorithms. It can be either explicitly specified, or include all algorithms supported by client library.
+ * Returns:
+ * - true if the compression string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or if compression is disabled, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+extern bool
+ zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Serialize the compressors array to string so it can be transmitted to the other side during the compression startup.
+ * For example, for array of two compressors (zstd, level 1), (zlib, level 2) resulting string would look like "zstd:1,zlib:2".
+ * Returns the resulting string.
+ */
+extern char
+ *zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors);
+
+/* Deserialize the compressors string received during the compression setup to a compressors array.
+ * For example, for string "zstd:1,zlib:2" compressors would be populated with 2 elements: (zstd, level 1), (zlib, level 2).
+ * Returns:
+ * - true if the compressors string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or string is empty, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+bool
+ zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors);
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index dd3e5efba3..774d4638bd 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -188,6 +188,9 @@ typedef struct Port
int keepalives_count;
int tcp_user_timeout;
+ char *compression_algorithms; /* Compression algorithms supported by
+ * client */
+
/*
* GSSAPI structures.
*/
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index f0786e08b4..d568a056bc 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -71,6 +71,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port *port);
extern int pq_getbytes(char *s, size_t len);
extern void pq_startmsgread(void);
extern void pq_endmsgread(void);
diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h
index ed26ad2256..21787e3006 100644
--- a/src/include/libpq/pqcomm.h
+++ b/src/include/libpq/pqcomm.h
@@ -137,6 +137,9 @@ typedef uint32 PacketLen;
extern bool Db_user_namespace;
+/* List of allowed compression algorithms */
+extern char *libpq_compress_algorithms;
+
/*
* In protocol 3.0 and later, the startup packet length is not fixed, but
* we set an arbitrary limit on it anyway. This is just to prevent simple
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 8217d0cb6b..0e4e457dd6 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -144,6 +144,12 @@ typedef struct PgBackendStatus
/* application name; MUST be null-terminated */
char *st_appname;
+ /* client-server traffic information */
+ uint64 st_rx_raw_bytes;
+ uint64 st_tx_raw_bytes;
+ uint64 st_rx_compressed_bytes;
+ uint64 st_tx_compressed_bytes;
+
/*
* Current command string; MUST be null-terminated. Note that this string
* possibly is truncated in the middle of a multi-byte character. As
@@ -301,6 +307,7 @@ extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
+extern void pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes);
extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
int buflen);
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 844c95d47d..1c11acc2c5 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,20 @@ endif
# The MSVC build system scrapes OBJS from this file. If you change any of
# the conditional additions of files to OBJS, update Mkvcbuild.pm to match.
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+SHLIB_LINK += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
+# We can't use Makefile variables here because the MSVC build system scrapes
+# OBJS from this file.
+
+
OBJS = \
$(WIN32RES) \
fe-auth-scram.o \
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index e8bcc88370..33ce6f2513 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -186,3 +186,6 @@ PQpipelineStatus 183
PQsetTraceFlags 184
PQmblenBounded 185
PQsendFlushRequest 186
+PQcompressor 187
+PQdecompressor 188
+PQreadPending 189
\ No newline at end of file
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 5fc16be849..d02b74b9a9 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -24,6 +24,7 @@
#include "common/ip.h"
#include "common/link-canary.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "common/string.h"
#include "fe-auth.h"
#include "libpq-fe.h"
@@ -339,6 +340,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "PGCOMPRESSION", "off", NULL,
+ "Libpq-compression", "", 16,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 15, /* sizeof("prefer-standby") = 15 */
@@ -447,6 +452,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zpqStream);
+ conn->zpqStream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -3260,11 +3269,14 @@ keep_going: /* We will come back to here until there is
}
/*
- * Validate message type: we expect only an authentication
- * request or an error here. Anything else probably means
- * it's not Postgres on the other end at all.
+ * Validate message type. We expect only:
+ * - authentication request ('R')
+ * - error ('E')
+ * - protocol compression acknowledgment ('z')
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
{
appendPQExpBuffer(&conn->errorMessage,
libpq_gettext("expected authentication request from server, but received %c\n"),
@@ -3342,6 +3354,60 @@ keep_going: /* We will come back to here until there is
return PGRES_POLLING_READING;
}
+ if (beresp == 'z') /* Switch on compression */
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ char *resp = malloc(msgLength);
+
+ pqGetnchar(resp, msgLength, conn);
+
+ if (!zpq_deserialize_compressors(resp, &compressors, &n_compressors))
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server returned unrecognized compression setting: %s\n"),
+ resp);
+ free(resp);
+ goto error_return;
+ }
+ free(resp);
+
+ if (n_compressors == 0)
+ {
+ /*
+ * If there are no compressors returned, it means that
+ * the server rejected all sent compression
+ * algorithms. Mark the data as consumed and do not
+ * enable the compression.
+ */
+ conn->inStart = conn->inCursor;
+ goto keep_going;
+ }
+
+ Assert(!conn->zpqStream);
+ conn->zpqStream = zpq_create(compressors, n_compressors,
+ (zpq_tx_func) pqsecure_write, (zpq_rx_func) pqsecure_read,
+ conn,
+ &conn->inBuffer[conn->inCursor],
+ conn->inEnd - conn->inCursor);
+ if (!conn->zpqStream)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("failed to initialize compression\n"));
+ free(compressors);
+ goto error_return;
+ }
+ /* reset buffer */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+
+ if (conn->n_compressors != 0 && beresp == 'v') /* NegotiateProtocolVersion */
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server does not support protocol compression\n"));
+ goto error_return;
+ }
+
/* Handle errors. */
if (beresp == 'E')
{
@@ -4085,6 +4151,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
@@ -6616,6 +6684,22 @@ PQuser(const PGconn *conn)
return conn->pguser;
}
+char *
+PQcompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_compress_algorithm_name(conn->zpqStream);
+}
+
+char *
+PQdecompressor(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+ return (char *) zpq_decompress_algorithm_name(conn->zpqStream);
+}
+
char *
PQpass(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 59121873d2..c5ec552354 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2010,9 +2010,7 @@ PQgetResult(PGconn *conn)
* EOF indication. We expect therefore that this won't result in any
* undue delay in reporting a previous write failure.)
*/
- if (flushResult ||
- pqWait(true, false, conn) ||
- pqReadData(conn) < 0)
+ if (flushResult || pqWait(true, false, conn) || pqReadData(conn) < 0)
{
/*
* conn->errorMessage has been set by pqWait or pqReadData. We
@@ -3785,6 +3783,12 @@ pqPipelineFlush(PGconn *conn)
return 0;
}
+int
+PQreadPending(PGconn *conn)
+{
+ return pqReadPending(conn);
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 7fcfe08fd2..1662a30b3e 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,12 +53,24 @@
#include "pg_config_paths.h"
#include "port/pg_bswap.h"
+#include <common/zpq_stream.h>
+
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
+/*
+ * Use zpq_read if compression is switched on
+ */
+#define pq_read_conn(conn) \
+ (conn->zpqStream \
+ ? zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd, false) \
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd))
+
/*
* PQlibVersion: return the libpq version number
*/
@@ -617,10 +629,17 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -712,10 +731,18 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
+
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -826,12 +853,18 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered_tx(conn->zpqStream))
{
int sent;
+ size_t processed = 0;
+ /*
+ * Use zpq_write if compression is switched on
+ */
+ sent = conn->zpqStream
+ ? zpq_write(conn->zpqStream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -839,8 +872,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+: pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +932,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered_tx(conn->zpqStream))
{
/*
* We didn't send it all, wait till we can send more.
@@ -964,7 +1000,7 @@ pqSendSome(PGconn *conn, int len)
int
pqFlush(PGconn *conn)
{
- if (conn->outCount > 0)
+ if (conn->outCount > 0 || zpq_buffered_tx(conn->zpqStream))
{
if (conn->Pfdebug)
fflush(conn->Pfdebug);
@@ -989,6 +1025,8 @@ pqFlush(PGconn *conn)
int
pqWait(int forRead, int forWrite, PGconn *conn)
{
+ if (forRead && conn->inCursor < conn->inEnd)
+ return 0;
return pqWaitTimed(forRead, forWrite, conn, (time_t) -1);
}
@@ -1046,6 +1084,9 @@ pqWriteReady(PGconn *conn)
*
* If SSL is in use, the SSL buffer is checked prior to checking the socket
* for read data directly.
+ *
+ * If ZPQ stream is in use, the ZPQ buffer is checked prior to checking
+ * the socket for read data directly.
*/
static int
pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
@@ -1061,14 +1102,10 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return -1;
}
-#ifdef USE_SSL
- /* Check for SSL library buffering read bytes */
- if (forRead && conn->ssl_in_use && pgtls_read_pending(conn))
+ if (forRead && (pqReadPending(conn) > 0))
{
- /* short-circuit the select */
return 1;
}
-#endif
/* We will retry as long as we get EINTR */
do
@@ -1088,6 +1125,33 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return result;
}
+/*
+ * Check if there is some data pending in ZPQ / SSL read buffers.
+ * Returns -1 on failure, 0 if no, 1 if yes.
+ */
+int
+pqReadPending(PGconn *conn)
+{
+ if (!conn)
+ return -1;
+
+ /* check for ZPQ stream buffered read bytes */
+ if (zpq_buffered_rx(conn->zpqStream))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+
+#ifdef USE_SSL
+ /* Check for SSL library buffering read bytes */
+ if (conn->ssl_in_use && pgtls_read_pending(conn))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+#endif
+ return 0;
+}
/*
* Check a file descriptor for read and/or write data, possibly waiting.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 26dbeaed97..e33a58088c 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -71,6 +71,15 @@ pqParseInput3(PGconn *conn)
*/
for (;;)
{
+ /*
+ * Read the buffered compressed data w/o blocking
+ */
+ if (conn->zpqStream && pqReadPending(conn) && (conn->inBufSize - conn->inEnd > 0)) {
+ int rc = zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, conn->inBufSize - conn->inEnd, true);
+ if (rc > 0) {
+ conn->inEnd += rc;
+ }
+ }
/*
* Try to read a message. First get the type code and length. Return
* if not enough data.
@@ -1708,7 +1717,7 @@ pqGetCopyData3(PGconn *conn, char **buffer, int async)
if (msgLength == 0)
{
/* Don't block if async read requested */
- if (async)
+ if (async && !pqReadPending(conn))
return 0;
/* Need to load more data */
if (pqWait(true, false, conn) ||
@@ -2174,6 +2183,50 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
return startpacket;
}
+/*
+ * Build comma-separated list of compression algorithms requested by client.
+ * It can be either explicitly specified by user in connection string, or
+ * include all algorithms supported by client library.
+ * This function returns true if the compression string is successfully parsed and
+ * stores a comma-separated list of algorithms in *client_compressors.
+ * If compression is disabled, then NULL is assigned to *client_compressors.
+ * Also it creates an array of compressor descriptors, each element of which corresponds to
+ * the corresponding algorithm name in *client_compressors list. This array is stored in PGconn
+ * and is used during handshake when a compression acknowledgment response is received from the server.
+ */
+static bool
+build_compressors_list(PGconn *conn, char **client_compressors, bool build_descriptors)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors))
+ {
+ return false;
+ }
+
+ *client_compressors = NULL;
+ if (build_descriptors)
+ {
+ conn->compressors = compressors;
+ conn->n_compressors = n_compressors;
+ }
+
+ if (n_compressors == 0)
+ {
+ /* no compressors available, return */
+ return true;
+ }
+
+ *client_compressors = zpq_serialize_compressors(compressors, n_compressors);
+
+ if (!build_descriptors)
+ {
+ free(compressors);
+ }
+ return true;
+}
+
/*
* Build a startup packet given a filled-in PGconn structure.
*
@@ -2220,6 +2273,18 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("replication", conn->replication);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
+ if (conn->compression && conn->compression[0])
+ {
+ char *client_compression_algorithms;
+
+ if (build_compressors_list((PGconn *) conn, &client_compression_algorithms, packet == NULL))
+ {
+ if (client_compression_algorithms)
+ {
+ ADD_STARTUP_OPTION("_pq_.compression", client_compression_algorithms);
+ }
+ }
+ }
if (conn->send_appname)
{
/* Use appname if present, otherwise use fallback */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 20eb855abc..51e807847d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -340,6 +340,8 @@ extern char *PQhostaddr(const PGconn *conn);
extern char *PQport(const PGconn *conn);
extern char *PQtty(const PGconn *conn);
extern char *PQoptions(const PGconn *conn);
+extern char *PQcompressor(const PGconn *conn);
+extern char *PQdecompressor(const PGconn *conn);
extern ConnStatusType PQstatus(const PGconn *conn);
extern PGTransactionStatusType PQtransactionStatus(const PGconn *conn);
extern const char *PQparameterStatus(const PGconn *conn,
@@ -498,6 +500,9 @@ extern PGPing PQpingParams(const char *const *keywords,
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
+extern int
+ PQreadPending(PGconn *conn);
+
/*
* "Fast path" interface --- not really recommended for application
* use
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index fcce13843e..47b9cdadbf 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "fe-auth-sasl.h"
#include "pqexpbuffer.h"
@@ -340,6 +341,7 @@ typedef struct pg_conn_host
* found in password file. */
} pg_conn_host;
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -392,6 +394,14 @@ struct pg_conn
* "sspi") */
char *ssl_min_protocol_version; /* minimum TLS protocol version */
char *ssl_max_protocol_version; /* maximum TLS protocol version */
+
+ char *compression; /* stream compression (boolean value, "any" or
+ * list of compression algorithms separated by
+ * comma) */
+ zpq_compressor *compressors; /* descriptors of compression algorithms
+ * chosen by client */
+ unsigned n_compressors; /* size of compressors array */
+
char *target_session_attrs; /* desired session properties */
/* Optional file to write trace info to */
@@ -572,6 +582,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream *zpqStream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
@@ -697,6 +710,7 @@ extern int pqWaitTimed(int forRead, int forWrite, PGconn *conn,
time_t finish_time);
extern int pqReadReady(PGconn *conn);
extern int pqWriteReady(PGconn *conn);
+extern int pqReadPending(PGconn *conn);
/* === in fe-secure.c === */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index b58b062b10..e4363bbf7c 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1765,7 +1765,7 @@ pg_stat_activity| SELECT s.datid,
s.query_id,
s.query,
s.backend_type
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_all_indexes| SELECT c.oid AS relid,
@@ -1877,8 +1877,14 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_auth AS gss_authenticated,
s.gss_princ AS principal,
s.gss_enc AS encrypted
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
+pg_stat_network_traffic| SELECT s.pid,
+ s.rx_raw_bytes,
+ s.tx_raw_bytes,
+ s.rx_compressed_bytes,
+ s.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
@@ -2047,7 +2053,7 @@ pg_stat_replication| SELECT s.pid,
w.sync_priority,
w.sync_state,
w.reply_time
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_replication_slots| SELECT s.slot_name,
@@ -2081,7 +2087,7 @@ pg_stat_ssl| SELECT s.pid,
s.ssl_client_dn AS client_dn,
s.ssl_client_serial AS client_serial,
s.ssl_issuer_dn AS issuer_dn
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ec3546d0c0..92dfb27a41 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -130,7 +130,7 @@ sub mkvcbuild
keywords.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c
rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c
- username.c wait_error.c wchar.c);
+ username.c wait_error.c wchar.c z_stream.c zpq_stream.c);
if ($solution->{options}->{openssl})
{
--
2.30.2
=
0002-Add-ZSTD-support.patch (application/octet-stream, 17.8K) download
From 258cbd4db6ff9ede34e42fa1818ab1927be039d6 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:59:01 +0500
Subject: [PATCH 2/3] Add ZSTD support
---
configure | 214 +++++++++++++++++++++++++++++++
configure.ac | 36 ++++++
src/Makefile.global.in | 1 +
src/backend/Makefile | 4 +
src/common/z_stream.c | 228 ++++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
src/interfaces/libpq/Makefile | 5 +
src/tools/msvc/Solution.pm | 1 +
8 files changed, 492 insertions(+)
diff --git a/configure b/configure
index 3f2aea0d7d..fbf93c1fbb 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,9 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -868,6 +871,7 @@ with_libxslt
with_system_tzdata
with_zlib
with_lz4
+with_zstd
with_gnu_ld
with_ssl
with_openssl
@@ -897,6 +901,8 @@ XML2_CFLAGS
XML2_LIBS
LZ4_CFLAGS
LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
LDFLAGS_EX
LDFLAGS_SL
PERL
@@ -1576,6 +1582,7 @@ Optional Packages:
use system time zone data in DIR
--without-zlib do not use Zlib
--with-lz4 build with LZ4 support
+ --with-zstd build with ZSTD support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -1605,6 +1612,8 @@ Some influential environment variables:
XML2_LIBS linker flags for XML2, overriding pkg-config
LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config
LZ4_LIBS linker flags for LZ4, overriding pkg-config
+ ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+ ZSTD_LIBS linker flags for ZSTD, overriding pkg-config
LDFLAGS_EX extra linker flags for linking executables only
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
@@ -9033,6 +9042,145 @@ fi
done
fi
+#
+# Zstd
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+
+$as_echo "#define HAVE_LIBZSTD 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd >= 1.0.0" >&5
+$as_echo_n "checking for libzstd >= 1.0.0... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+ pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+ pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ else
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$ZSTD_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (libzstd >= 1.0.0) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+ ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -12523,6 +12671,58 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_decompressStream in -lzstd" >&5
+$as_echo_n "checking for ZSTD_decompressStream in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_decompressStream+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_decompressStream ();
+int
+main ()
+{
+return ZSTD_decompressStream ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_decompressStream=yes
+else
+ ac_cv_lib_zstd_ZSTD_decompressStream=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_decompressStream" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_decompressStream" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_decompressStream" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13854,6 +14054,20 @@ fi
done
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index 95287705f6..7759ff3613 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,28 @@ if test "$with_lz4" = yes; then
done
fi
+#
+# Zstd
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support], [AC_DEFINE([HAVE_LIBZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+ PKG_CHECK_MODULES(ZSTD, [libzstd >= 1.0.0])
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -1242,6 +1264,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_decompressStream, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1488,6 +1517,13 @@ if test "$with_lz4" = yes; then
AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 79ed589ce9..71e14ee68f 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -197,6 +197,7 @@ with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
with_lz4 = @with_lz4@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index dab379b412..94cae64831 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,10 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
ifeq ($(with_lz4),yes)
LIBS += -llz4
endif
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index d04f22b924..13ff7cfb30 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -89,6 +89,231 @@ struct ZStream
bool not_flushed;
};
+#if HAVE_LIBZSTD
+
+#include <stdlib.h>
+#include <zstd.h>
+
+/*
+ * Maximum allowed back-reference distance, expressed as power of 2.
+ * This setting controls max compressor/decompressor window size.
+ * More details https://github.com/facebook/zstd/blob/v1.4.7/lib/zstd.h#L536
+ */
+#define ZSTD_WINDOWLOG_LIMIT 23 /* set max window size to 8MB */
+
+
+typedef struct ZS_ZSTD_CStream
+{
+ ZSTD_CStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_CStream;
+
+typedef struct ZS_ZSTD_DStream
+{
+ ZSTD_DStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_DStream;
+
+static void *
+zstd_create_compressor(int level)
+{
+ size_t rc;
+ ZS_ZSTD_CStream *c_stream = (ZS_ZSTD_CStream *) malloc(sizeof(ZS_ZSTD_CStream));
+
+ c_stream->stream = ZSTD_createCStream();
+ rc = ZSTD_initCStream(c_stream->stream, level);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeCStream(c_stream->stream);
+ free(c_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_CCtx_setParameter(c_stream->stream, ZSTD_c_windowLog, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ c_stream->error = NULL;
+ return c_stream;
+}
+
+static void *
+zstd_create_decompressor()
+{
+ size_t rc;
+ ZS_ZSTD_DStream *d_stream = (ZS_ZSTD_DStream *) malloc(sizeof(ZS_ZSTD_DStream));
+
+ d_stream->stream = ZSTD_createDStream();
+ rc = ZSTD_initDStream(d_stream->stream);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeDStream(d_stream->stream);
+ free(d_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_DCtx_setParameter(d_stream->stream, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ d_stream->error = NULL;
+ return d_stream;
+}
+
+static ssize_t
+zstd_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+ size_t rc;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ rc = ZSTD_decompressStream(ds->stream, &out, &in);
+
+ *src_processed = in.pos;
+ *dst_processed = out.pos;
+ if (ZSTD_isError(rc))
+ {
+ ds->error = ZSTD_getErrorName(rc);
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ if (rc == 0)
+ {
+ return ZS_STREAM_END;
+ }
+
+ if (out.pos == out.size)
+ {
+ /*
+ * if `output.pos == output.size`, there might be some data left
+ * within internal buffers
+ */
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ if (in.pos < src_size) /* Has something to compress in input buffer */
+ {
+ size_t rc = ZSTD_compressStream(cs->stream, &out, &in);
+
+ *dst_processed = out.pos;
+ *src_processed = in.pos;
+ if (ZSTD_isError(rc))
+ {
+ cs->error = ZSTD_getErrorName(rc);
+ return ZS_COMPRESS_ERROR;
+ }
+ }
+
+ if (in.pos == src_size) /* All data is compressed: flush internal zstd
+ * buffer */
+ {
+ size_t tx_not_flushed = ZSTD_flushStream(cs->stream, &out);
+
+ *dst_processed = out.pos;
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ size_t tx_not_flushed;
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_outBuffer output;
+
+ output.dst = dst;
+ output.pos = 0;
+ output.size = dst_size;
+
+ do
+ {
+ tx_not_flushed = ZSTD_endStream(cs->stream, &output);
+ } while ((tx_not_flushed > 0) && (output.pos < output.size));
+
+ *dst_processed = output.pos;
+
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static void
+zstd_free_compressor(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ ZSTD_freeCStream(cs->stream);
+ free(cs);
+ }
+}
+
+static void
+zstd_free_decompressor(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ ZSTD_freeDStream(ds->stream);
+ free(ds);
+ }
+}
+
+static char const *
+zstd_compress_error(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ return cs->error;
+}
+
+static char const *
+zstd_decompress_error(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ return ds->error;
+}
+
+static char const *
+zstd_name(void)
+{
+ return "zstd";
+}
+
+#endif
+
#if HAVE_LIBZ
#include <stdlib.h>
@@ -417,6 +642,9 @@ no_compression_name(void)
*/
static ZAlgorithm const zs_algorithms[] =
{
+#if HAVE_LIBZSTD
+ {zstd_name, zstd_create_compressor, zstd_create_decompressor, zstd_decompress, zstd_compress, zstd_free_compressor, zstd_free_decompressor, zstd_compress_error, zstd_decompress_error, zstd_end},
+#endif
#if HAVE_LIBZ
{zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
#endif
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9d9bd6b9ef..52c66d2e61 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 1c11acc2c5..00be5ec1cc 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -34,6 +34,11 @@ LIBS += -llz4
SHLIB_LINK += -llz4
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
ifeq ($(with_zlib),yes)
LIBS += -lz
SHLIB_LINK += -lz
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index e47c2d648c..e402e3af15 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
--
2.30.2
=
0003-Turn-on-zlib-compression-for-CI-test-runs.patch (application/octet-stream, 3.7K) download
From 96274f45d5ded8e0d00e3ff59ffb7c046501e99c Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Wed, 12 Jan 2022 23:14:02 +0500
Subject: [PATCH 3/3] Turn on zlib compression for CI test runs
---
src/backend/utils/misc/guc.c | 2 +-
src/common/zpq_stream.c | 34 ++++++++-----------------------
src/interfaces/libpq/fe-connect.c | 2 +-
3 files changed, 11 insertions(+), 27 deletions(-)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fdf77e383d..74e2f617ad 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -4589,7 +4589,7 @@ static struct config_string ConfigureNamesString[] =
NULL
},
&libpq_compress_algorithms,
- "off",
+ "zlib",
check_libpq_compression, NULL, NULL
},
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
index c72b434c5d..69f1b4bba9 100644
--- a/src/common/zpq_stream.c
+++ b/src/common/zpq_stream.c
@@ -156,21 +156,8 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
for (i = 0; i < 256; i++)
{
- zpq->compressor_by_msg_type[i] = -1;
- }
-
- for (i = 0; i < zpq->n_compressors; i++)
- {
- /* compress CopyData, DataRow and Query messages */
- if (zpq->compressor_by_msg_type['d'] == -1) {
- zpq->compressor_by_msg_type['d'] = i;
- }
- if (zpq->compressor_by_msg_type['D'] == -1) {
- zpq->compressor_by_msg_type['D'] = i;
- }
- if (zpq->compressor_by_msg_type['Q'] == -1) {
- zpq->compressor_by_msg_type['Q'] = i;
- }
+ /* Compress all messages during the CI test runs */
+ zpq->compressor_by_msg_type[i] = 0;
}
}
@@ -183,11 +170,8 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
static inline int
zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
{
- if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
- {
- return zpq->compressor_by_msg_type[(unsigned char) msg_type];
- }
- return -1;
+ /* Compress messages with any length during the CI test runs */
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
}
/*
@@ -197,7 +181,7 @@ zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
static inline bool
zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
{
- return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
+ return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
}
/*
@@ -990,7 +974,7 @@ zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressor
/*
* single entry looks like "alg_name:compression_level," so +2 is for
- * ":" and "," symbols (or trailing null)
+ * ":" and "," symbols (or trailing null)
*/
total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
}
@@ -999,9 +983,9 @@ zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressor
for (i = 0; i < n_compressors; i++)
{
- p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
- if (i < n_compressors - 1)
- *p++ = ',';
+ p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
+ if (i < n_compressors - 1)
+ *p++ = ',';
}
return res;
}
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index d02b74b9a9..7c7a0c1a3b 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -340,7 +340,7 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
- {"compression", "PGCOMPRESSION", "off", NULL,
+ {"compression", "PGCOMPRESSION", "zlib", NULL,
"Libpq-compression", "", 16,
offsetof(struct pg_conn, compression)},
--
2.30.2
=
* Re: libpq compression (part 2)
@ 2022-01-13 23:58 Justin Pryzby <pryzby@telsasoft.com>
parent: Daniil Zakhlystov <usernamedt@yandex-team.ru>
0 siblings, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-01-13 23:58 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; +Cc: pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On Fri, Jan 14, 2022 at 02:12:17AM +0500, Daniil Zakhlystov wrote:
> Hi, Justin!
>
> First of all, thanks for the detailed review. I’ve applied your patches to the current version.
Note that my message had other comments that weren't addressed in this patch.
Your 0003 patch has a couple "noise" hunks that get rid of ^M characters added
in previous patches. The ^M shouldn't be added in the first place. Did you
apply my fixes using git-am or something else ?
On Fri, Jan 14, 2022 at 02:12:17AM +0500, Daniil Zakhlystov wrote:
> > On 12 Jan 2022, at 19:15, Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > zlib still causes check-world to get stuck. I first mentioned this last March:
> > 20210319062800.GI11765@telsasoft.com
> >
...
> > I removed the thread from the CFBOT until that's resolved.
>
> I’ve fixed the failing tests, now they should pass.
macos: passed
linux: timed out after 1hr
freebsd: failed in pg_rewind: ... <= replay_lsn AND state = 'streaming' FROM ...
windows: "Failed test 'data_checksums=on is reported on an offline cluster stdout /(?^:^on$)/'" / WARNING: 01000: algorithm zlib is not supported
Note that it's possible and easy to kick off a CI run using any github account:
see ./src/tools/ci/README
For me, it's faster than running check-world -j4 locally, and runs tests on 4
OSes.
I re-ran your branch under my own account and linux didn't get stuck (and the
compiler warnings tests passed). But on a third attempt, macos failed the
pg_rewind test, and bsd failed the subscription test:
| SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');
--
Justin
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-01-17 21:06 Daniil Zakhlystov <usernamedt@yandex-team.ru>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 27+ messages in thread
From: Daniil Zakhlystov @ 2022-01-17 21:06 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
Hi!
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> +/* GUC variable containing the allowed compression algorithms list (separated by comma) */
> +char *libpq_compress_algorithms;
>
> => The global variable is conventionally initialized to the same default as
> guc.c (even though the GUC default value will be applied at startup).
I’ve updated libpq_compress_algorithms to match the corresponding guc.c setting value.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> + * - NegotiateProtocolVersion in cases when server does not support protocol compression
> + * Anything else probably means it's not Postgres on the other end at all.
> */
> - if (!(beresp == 'R' || beresp == 'E'))
> + if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
>
> => I think NegotiateProtocolVersion and 'v' are from an old version of the
> patch and no longer used ?
No, it is still relevant in the current patch version. I’ve added more comments regarding this case
and also made more transparent compression negotiation behavior.
If the client requested the compression, but the server does not support the libpq compression feature
because of the old Postgres version, the client will report an error and exit. If the server rejected
the client's compression request (for example, if libpq_compression is set to ‘off’ or no matching
algorithms found), the client will also report an error and exit.
> On 14 Jan 2022, at 04:58, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> Your 0003 patch has a couple "noise" hunks that get rid of ^M characters added
> in previous patches. The ^M shouldn't be added in the first place. Did you
> apply my fixes using git-am or something else ?
I’ve fixed the line endings and applied the pgindent. Now each patch should be OK.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> + * Report current transaction start timestamp as the specified value.
> + * Zero means there is no active transaction.
>
> => The comment doesn't correspond to the function
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> + return id >= 0 && id < (sizeof(zs_algorithms) / sizeof(*zs_algorithms));
> + size_t n_algorithms = sizeof(zs_algorithms) / sizeof(*zs_algorithms);
>
> => You can use lengthof() for these
Thanks, I’ve updated the patch and corrected the highlighted issues.
> On 1 Jan 2022, at 22:25, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> I think the psql conninfo Compressor/Decompressor line may be confusing.
> Maybe it should talk about Client->Server and Server->Client.
> Maybe it should be displayed on a single line.
>
> Actually, right now the \conninfo part is hardly useful, since the compressor
> is allocated lazily/"on demand", so it shows the compression of some previous
> command, but maybe not the last command, and maybe not the next command…
I’ve updated the \conninfo part, now it shows the list of the negotiated compression algorithms.
Also, I’ve added the check for the compression option to the frontend side.
> On 14 Jan 2022, at 04:58, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> macos: passed
> linux: timed out after 1hr
> freebsd: failed in pg_rewind: ... <= replay_lsn AND state = 'streaming' FROM ...
> windows: "Failed test 'data_checksums=on is reported on an offline cluster stdout /(?^:^on$)/'" / WARNING: 01000: algorithm zlib is not supported
>
> Note that it's possible and easy to kick off a CI run using any github account:
> see ./src/tools/ci/README
>
> For me, it's faster than running check-world -j4 locally, and runs tests on 4
> OSes.
I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
CI for some reason, but I guess it happens because of my GitHub account limitation.
--
Thanks,
Daniil Zakhlystov
Attachments: download all (3 files, .tar.gz)
0001-Implement-libpq-compression.patch (application/octet-stream, 117.5K) download
From c256d1282d5646fee476188a4c1da7d0e3aef5ac Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Tue, 18 Jan 2022 00:51:05 +0500
Subject: [PATCH 1/3] Implement libpq compression
---
.../postgres_fdw/expected/postgres_fdw.out | 2 +-
doc/src/sgml/config.sgml | 21 +
doc/src/sgml/libpq.sgml | 37 +
doc/src/sgml/protocol.sgml | 174 +++
src/Makefile.global.in | 1 +
src/backend/Makefile | 8 +
src/backend/catalog/system_views.sql | 9 +
src/backend/libpq/pqcomm.c | 277 ++++-
src/backend/postmaster/postmaster.c | 10 +
.../libpqwalreceiver/libpqwalreceiver.c | 13 +-
src/backend/utils/activity/backend_status.c | 29 +
src/backend/utils/adt/pgstatfuncs.c | 50 +-
src/backend/utils/misc/guc.c | 30 +
src/bin/pgbench/pgbench.c | 17 +-
src/bin/psql/command.c | 17 +
src/common/Makefile | 4 +-
src/common/z_stream.c | 669 +++++++++++
src/common/zpq_stream.c | 1022 +++++++++++++++++
src/include/catalog/pg_proc.dat | 18 +-
src/include/common/z_stream.h | 109 ++
src/include/common/zpq_stream.h | 120 ++
src/include/libpq/libpq-be.h | 3 +
src/include/libpq/libpq.h | 1 +
src/include/libpq/pqcomm.h | 3 +
src/include/utils/backend_status.h | 7 +
src/interfaces/libpq/Makefile | 14 +
src/interfaces/libpq/exports.txt | 2 +
src/interfaces/libpq/fe-connect.c | 125 +-
src/interfaces/libpq/fe-exec.c | 10 +-
src/interfaces/libpq/fe-misc.c | 92 +-
src/interfaces/libpq/fe-protocol3.c | 71 +-
src/interfaces/libpq/libpq-fe.h | 4 +
src/interfaces/libpq/libpq-int.h | 14 +
src/test/regress/expected/rules.out | 14 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
35 files changed, 2903 insertions(+), 96 deletions(-)
create mode 100644 src/common/z_stream.c
create mode 100644 src/common/zpq_stream.c
create mode 100644 src/include/common/z_stream.h
create mode 100644 src/include/common/zpq_stream.h
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 7d6f7d9e3d..0d30cfc4aa 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -9509,7 +9509,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
+HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, sslcrldir, sslsni, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, compression, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, truncatable, fetch_size, batch_size, async_capable, keep_connections
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4cd9818acf..6b4a1b63c4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1035,6 +1035,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-libpq-compression" xreflabel="libpq_compression">
+ <term><varname>libpq_compression</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>libpq_compression</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter controls the available client-server traffic compression methods.
+ It allows rejecting compression requests even if it is supported by the server (for example, due to security, or CPU consumption).
+ The default is <literal>on</literal>, which means that all supported compression methods are allowed.
+ For more precise control, a list of the allowed compression methods can be specified.
+ For example, to allow only <literal>lz4</literal> and <literal>zlib</literal>, set the setting to <literal>lz4,zlib</literal>.
+ Also, maximal allowed compression level can be specified for each method, e.g. <literal>lz4:1,zlib:2</literal> setting will set the
+ maximal compression level for <literal>lz4</literal> to 1 and <literal>zlib</literal> to 2.
+ If a client requests the compression with a higher compression level, it will be set to the maximal allowed one.
+ Default (and recommended) maximal compression level for each algorithm is 1.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 14f35d37f6..5a16b4b64b 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1260,6 +1260,43 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. The client sends a request with
+ a list of compression algorithms. Compression can be requested by a client
+ by including the <literal>"compression"</literal> option in its connection string.
+ This can either be a boolean value to enable or disable compression
+ (<literal>"true"</literal>/<literal>"false"</literal>,
+ <literal>"on"</literal>/<literal>"off"</literal>,
+ <literal>"yes"</literal>/<literal>"no"</literal>,
+ <literal>"1"</literal>/<literal>"0"</literal>),
+ <literal>"any"</literal>,
+ or an explicit list of comma-separated compression algorithms
+ which can optionally include compression level (<literal>"zlib,lz4:2"</literal>).
+ If compression is enabled but an algorithm is not explicitly specified,
+ the client library sends its full list of supported algorithms.
+ The server intersects the received compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ </para>
+ <para>
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ Compressed data is transmitted via the CompressedData messages.
+ </para>
+ <para>
+ Support for compression algorithms must be enabled when the server is compiled.
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option). In both cases, streaming mode is used.
+ By default, compression is not requested by the client.
+ Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 34a7034282..332041030a 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ It is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especially useful for importing/exporting data to/from the database using the <literal>COPY</literal> command
+ and for replication (both physical and logical). Compression can also reduce the server's response time
+ for queries returning a large amount of data (for example, JSON, BLOBs, text, ...).
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option).
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -262,6 +271,21 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CompressionAck</term>
+ <listitem>
+ <para>
+ The server accepts the client's compression request.
+ Compression is requested when a client connection includes the "compression" option, which contains a list of the requested compression algorithms.
+ The server intersects the requested compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>AuthenticationOk</term>
<listitem>
@@ -3577,6 +3601,141 @@ following:
</varlistentry>
+<varlistentry>
+<term>
+CompressionAck (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ List of the negotiated compression algorithms.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
+<varlistentry>
+<term>
+SetCompressionMethod (F & B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('k')
+</term>
+<listitem>
+<para>
+ Switch the current compression algorithm. Following CompressedData messages will be compressed via the compressed algorithm specified in this message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32(5)
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ Index of the new compression algorithm.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+CompressedData (F & B)
+</term>
+<listitem>
+<para>
+
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('m')
+</term>
+<listitem>
+<para>
+ Identifies the message as compressed data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ Compressed message data.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
<varlistentry>
<term>
AuthenticationSASLContinue (B)
@@ -6142,6 +6301,21 @@ StartupMessage (F)
</para>
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+ <literal>_pq_.compression</literal>
+</term>
+<listitem>
+<para>
+ Request compression of libpq traffic. The value is a list of compression algorithms requested by the client with an optional
+ specification of compression level: <literal>"zlib,lz4:5"</literal>.
+ If the server does not support compression, the backend will ignore the _pq_.compression
+ parameter and will not send the CompressionAck message to the frontend.
+ By default, compression is disabled. Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>.
+</para>
+</listitem>
+</varlistentry>
</variablelist>
In addition to the above, other parameters may be listed.
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 05c54b27de..79ed589ce9 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_lz4 = @with_lz4@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index add9560be4..dab379b412 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..a9bdd8ee99 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -846,6 +846,15 @@ CREATE VIEW pg_stat_activity AS
LEFT JOIN pg_database AS D ON (S.datid = D.oid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
+CREATE VIEW pg_stat_network_traffic AS
+ SELECT
+ S.pid,
+ S.rx_raw_bytes,
+ S.tx_raw_bytes,
+ S.rx_compressed_bytes,
+ S.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL) AS S;
+
CREATE VIEW pg_stat_replication AS
SELECT
S.pid,
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index f05723dc92..ea813bfe26 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -60,6 +60,7 @@
#include <signal.h>
#include <fcntl.h>
#include <grp.h>
+#include <pgstat.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/socket.h>
@@ -77,11 +78,14 @@
#include "common/ip.h"
#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
#include "miscadmin.h"
#include "port/pg_bswap.h"
#include "storage/ipc.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/builtins.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -108,6 +112,9 @@
int Unix_socket_permissions;
char *Unix_socket_group;
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms = "off";
+
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -130,6 +137,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream * PqStream;
+
+
/*
* Message status
*/
@@ -167,6 +177,145 @@ const PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+static ssize_t
+write_compressed(void *arg, void const *data, size_t size)
+{
+ ssize_t rc = secure_write((Port *) arg, (void *) data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, 0, rc);
+ return rc;
+}
+
+static ssize_t
+read_compressed(void *arg, void *data, size_t size)
+{
+ ssize_t rc = secure_read((Port *) arg, data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, rc, 0);
+ return rc;
+}
+
+/*
+ * Send the chosen compression algorithms to the client
+ */
+static void
+SendCompressionACK(char *compressors_str)
+{
+ StringInfoData buf;
+
+ pq_beginmessage(&buf, 'z');
+ pq_sendstring(&buf, compressors_str);
+ pq_endmessage(&buf);
+ pq_flush();
+}
+
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port *port)
+{
+ zpq_compressor *server_compressors;
+ zpq_compressor *client_compressors;
+ zpq_compressor *res_compressors;
+ size_t n_client_compressors;
+ size_t n_server_compressors;
+ size_t n_res_compressors = 0;
+ char *res_compressors_str;
+ char *client_compression_algorithms = port->compression_algorithms;
+
+ if (!client_compression_algorithms || !libpq_compress_algorithms)
+ {
+ return 0;
+ }
+
+ if (zpq_parse_compression_setting(libpq_compress_algorithms, &server_compressors, &n_server_compressors) == -1)
+ {
+ ereport(LOG, errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms));
+ return 0;
+ }
+
+ if (n_server_compressors == 0)
+ {
+ /*
+ * No enabled server compressors available, abort the compression
+ * initialization.
+ */
+ /* Send the compression acknowledgment with empty compressors list. */
+ char *empty_response = "";
+
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ if (!zpq_deserialize_compressors(client_compression_algorithms, &client_compressors, &n_client_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse received client compression methods: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ if (n_client_compressors == 0)
+ {
+ /*
+ * client did not provide any compression algorithms that server is
+ * compiled to support
+ */
+ ereport(LOG, (errmsg("server doesn't support any of the compression methods requested by the client: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ res_compressors = malloc(Max(n_client_compressors, n_server_compressors) * sizeof(zpq_compressor));
+
+ /*
+ * Intersect client and server compressors to determine the final list of
+ * the supported compressors. O(N^2) is negligible because of a small
+ * number of the compression methods.
+ */
+ for (size_t i = 0; i < n_client_compressors; i++)
+ {
+ for (size_t j = 0; j < n_server_compressors; j++)
+ {
+ if (client_compressors[i].impl == server_compressors[j].impl)
+ {
+ res_compressors[n_res_compressors].impl = client_compressors[i].impl;
+ /* prefer the lower compression level */
+ res_compressors[n_res_compressors++].level = Min(client_compressors[i].level, server_compressors[j].level);
+ break;
+ }
+ }
+ }
+ free(client_compressors);
+ free(server_compressors);
+
+ if (n_res_compressors == 0)
+ {
+ char *empty_response = "";
+
+ ereport(LOG, (errmsg("did not find any matches between the client and server compression methods, not enabling compression")));
+ free(res_compressors);
+ /* send the compression ack with empty compressors list */
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ res_compressors_str = zpq_serialize_compressors(res_compressors, n_res_compressors);
+ SendCompressionACK(res_compressors_str);
+
+ /* Init compression */
+ PqStream = zpq_create(res_compressors, n_res_compressors, write_compressed, read_compressed, MyProcPort, NULL, 0);
+ if (!PqStream)
+ {
+ ereport(LOG, (errmsg("failed to initialize the compression stream")));
+ return -1;
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -270,6 +419,9 @@ socket_close(int code, Datum arg)
}
#endif /* ENABLE_GSS */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -941,12 +1093,15 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * nowait parameter toggles non-blocking mode.
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -962,21 +1117,40 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ /*
+ * If streaming compression is enabled then use correspondent
+ * compression read function.
+ */
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, false)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
if (r < 0)
{
+ if (r == ZS_DECOMPRESS_ERROR)
+ {
+ char const *msg = zpq_decompress_error(PqStream);
+
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -997,7 +1171,8 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ pgstat_report_network_traffic(r, 0, 0, 0);
+ return r;
}
}
@@ -1012,7 +1187,8 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -1031,7 +1207,8 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1048,48 +1225,15 @@ pq_peekbyte(void)
int
pq_getbyte_if_available(unsigned char *c)
{
- int r;
+ int r = 0;
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1110,7 +1254,8 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1144,7 +1289,8 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1168,7 +1314,6 @@ pq_buffer_has_data(void)
return (PqRecvPointer < PqRecvLength);
}
-
/* --------------------------------
* pq_startmsgread - begin reading a message from the client.
*
@@ -1372,13 +1517,24 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered_tx(PqStream) != 0)
+
+ /*
+ * has more data to flush or unsent data in internal compression
+ * buffer
+ */
{
int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
- if (r <= 0)
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1421,12 +1577,12 @@ internal_flush(void)
InterruptPending = 1;
return EOF;
}
+ pgstat_report_network_traffic(0, r, 0, 0);
last_reported_send_errno = 0; /* reset after any successful send */
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
@@ -1443,7 +1599,7 @@ socket_flush_if_writable(void)
int res;
/* Quick exit if nothing to do */
- if (PqSendPointer == PqSendStart)
+ if ((PqSendPointer == PqSendStart) && (zpq_buffered_tx(PqStream) == 0))
return 0;
/* No-op if reentrant call */
@@ -1466,7 +1622,7 @@ socket_flush_if_writable(void)
static bool
socket_is_send_pending(void)
{
- return (PqSendStart < PqSendPointer);
+ return (PqSendStart < PqSendPointer || (zpq_buffered_tx(PqStream) != 0));
}
/* --------------------------------
@@ -1989,3 +2145,16 @@ pq_check_connection(void)
return true;
}
+
+PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+
+Datum
+pg_compression_algorithm(PG_FUNCTION_ARGS)
+{
+ char const *algorithm_name = PqStream ? zpq_compress_algorithm_name(PqStream) : NULL;
+
+ if (algorithm_name)
+ PG_RETURN_TEXT_P(cstring_to_text(algorithm_name));
+ else
+ PG_RETURN_NULL();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index dc4afdd75a..39dedff751 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2233,6 +2233,8 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "_pq_.compression") == 0)
+ port->compression_algorithms = pstrdup(valptr);
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4555,6 +4557,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0d89db4e6a..8ab5141cda 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -203,11 +203,14 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
else
io_flag = WL_SOCKET_WRITEABLE;
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
- PQsocket(conn->streamConn),
- 0,
- WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
+ if (status == PGRES_POLLING_READING && PQreadPending(conn->streamConn))
+ rc = WL_SOCKET_READABLE;
+ else
+ rc = WaitLatchOrSocket(MyLatch,
+ WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
+ PQsocket(conn->streamConn),
+ 0,
+ WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
/* Interrupted? */
if (rc & WL_LATCH_SET)
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2fecf26a2c..2644dee670 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -338,6 +338,9 @@ pgstat_bestart(void)
lbeentry.st_xact_start_timestamp = 0;
lbeentry.st_databaseid = MyDatabaseId;
+ lbeentry.st_tx_raw_bytes = lbeentry.st_rx_raw_bytes =
+ lbeentry.st_tx_compressed_bytes = lbeentry.st_rx_compressed_bytes = 0;
+
/* We have userid for client-backends, wal-sender and bgworker processes */
if (lbeentry.st_backendType == B_BACKEND
|| lbeentry.st_backendType == B_WAL_SENDER
@@ -716,6 +719,32 @@ pgstat_report_xact_timestamp(TimestampTz tstamp)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Report network raw or compressed tx/rx traffic as the specified values.
+ */
+void
+pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ if (!pgstat_track_activities || !beentry)
+ return;
+
+ /*
+ * Update my status entry, following the protocol of bumping
+ * st_changecount before and after. We use a volatile pointer here to
+ * ensure the compiler doesn't try to get cute.
+ */
+ PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
+
+ beentry->st_rx_raw_bytes += rx_raw_bytes;
+ beentry->st_tx_raw_bytes += tx_raw_bytes;
+ beentry->st_rx_compressed_bytes += rx_compressed_bytes;
+ beentry->st_tx_compressed_bytes += tx_compressed_bytes;
+
+ PGSTAT_END_WRITE_ACTIVITY(beentry);
+}
+
/* ----------
* pgstat_read_current_status() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 15cb17ace4..cff5dbf420 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -567,7 +567,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
Datum
pg_stat_get_activity(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_ACTIVITY_COLS 30
+#define PG_STAT_GET_ACTIVITY_COLS 34
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
int pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
@@ -916,6 +916,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[29] = true;
else
values[29] = UInt64GetDatum(beentry->st_query_id);
+ values[30] = beentry->st_rx_raw_bytes;
+ values[31] = beentry->st_tx_raw_bytes;
+ values[32] = beentry->st_rx_compressed_bytes;
+ values[33] = beentry->st_tx_compressed_bytes;
}
else
{
@@ -944,6 +948,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[27] = true;
nulls[28] = true;
nulls[29] = true;
+ nulls[30] = true;
+ nulls[31] = true;
+ nulls[32] = true;
+ nulls[33] = true;
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -1143,6 +1151,46 @@ pg_stat_get_backend_start(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(result);
}
+Datum
+pg_stat_get_network_traffic(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_NETWORK_TRAFFIC_COLS 4
+ TupleDesc tupdesc;
+ Datum values[PG_STAT_NETWORK_TRAFFIC_COLS];
+ bool nulls[PG_STAT_NETWORK_TRAFFIC_COLS];
+ int32 beid = PG_GETARG_INT32(0);
+ PgBackendStatus *beentry;
+
+ if ((beentry = pgstat_fetch_stat_beentry(beid)) == NULL)
+ PG_RETURN_NULL();
+ else if (!HAS_PGSTAT_PERMISSIONS(beentry->st_userid))
+ PG_RETURN_NULL();
+
+ /* Initialise values and NULL flags arrays */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(PG_STAT_NETWORK_TRAFFIC_COLS);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "rx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "tx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "rx_compressed_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "tx_compressed_bytes",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ /* Fill values and NULLs */
+ values[0] = beentry->st_rx_raw_bytes;
+ values[1] = beentry->st_tx_raw_bytes;
+ values[2] = beentry->st_rx_compressed_bytes;
+ values[3] = beentry->st_tx_compressed_bytes;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
Datum
pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 4c94f09c64..9d8fd9b817 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -52,6 +52,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "common/string.h"
+#include "common/zpq_stream.h"
#include "funcapi.h"
#include "jit/jit.h"
#include "libpq/auth.h"
@@ -192,6 +193,7 @@ static bool check_stage_log_stats(bool *newval, void **extra, GucSource source);
static bool check_log_stats(bool *newval, void **extra, GucSource source);
static bool check_canonical_path(char **newval, void **extra, GucSource source);
static bool check_timezone_abbreviations(char **newval, void **extra, GucSource source);
+static bool check_libpq_compression(char **newval, void **extra, GucSource source);
static void assign_timezone_abbreviations(const char *newval, void *extra);
static void pg_timezone_abbrev_initialize(void);
static const char *show_archive_command(void);
@@ -4593,6 +4595,16 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"libpq_compression", PGC_SIGHUP, CLIENT_CONN_OTHER,
+ gettext_noop("Sets the list of allowed libpq compression algorithms."),
+ NULL
+ },
+ &libpq_compress_algorithms,
+ "off",
+ check_libpq_compression, NULL, NULL
+ },
+
{
{"application_name", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Sets the application name to be reported in statistics and logs."),
@@ -5025,6 +5037,8 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
@@ -11859,6 +11873,22 @@ check_ssl(bool *newval, void **extra, GucSource source)
return true;
}
+static bool
+check_libpq_compression(char **newval, void **extra, GucSource source)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (zpq_parse_compression_setting(*newval, &compressors, &n_compressors) == -1)
+ {
+ GUC_check_errdetail("Cannot parse the libpq_compression setting.");
+ return false;
+ }
+
+ free(compressors);
+ return true;
+}
+
static bool
check_stage_log_stats(bool *newval, void **extra, GucSource source)
{
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97f2a1f80a..554835e08e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6661,6 +6661,9 @@ threadRun(void *arg)
int nsocks; /* number of sockets to be waited for */
pg_time_usec_t min_usec;
pg_time_usec_t now = 0; /* set this only if needed */
+ bool buffered_rx = false; /* true if some of the clients has
+ * data left in SSL/ZPQ read
+ * buffers */
/*
* identify which client sockets should be checked for input, and
@@ -6695,6 +6698,9 @@ threadRun(void *arg)
*/
int sock = PQsocket(st->con);
+ /* check if conn has buffered SSL / ZPQ read data */
+ buffered_rx = buffered_rx || PQreadPending(st->con);
+
if (sock < 0)
{
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
@@ -6738,7 +6744,7 @@ threadRun(void *arg)
{
if (nsocks > 0)
{
- rc = wait_on_socket_set(sockets, min_usec);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, min_usec);
}
else /* nothing active, simple sleep */
{
@@ -6747,7 +6753,7 @@ threadRun(void *arg)
}
else /* no explicit delay, wait without timeout */
{
- rc = wait_on_socket_set(sockets, 0);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, 0);
}
if (rc < 0)
@@ -6786,8 +6792,11 @@ threadRun(void *arg)
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
goto done;
}
-
- if (!socket_has_input(sockets, sock, nsocks++))
+ if (PQreadPending(st->con))
+ {
+ nsocks++;
+ }
+ else if (!socket_has_input(sockets, sock, nsocks++))
continue;
}
else if (st->state == CSTATE_FINISHED ||
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index f590474855..211596077e 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -171,6 +171,7 @@ static void print_with_linenumbers(FILE *output, char *lines,
const char *header_keyword);
static void minimal_error_message(PGresult *res);
+static void printCompressionInfo(void);
static void printSSLInfo(void);
static void printGSSInfo(void);
static bool printPsetInfo(const char *param, printQueryOpt *popt);
@@ -640,6 +641,7 @@ exec_command_conninfo(PsqlScanState scan_state, bool active_branch)
printf(_("You are connected to database \"%s\" as user \"%s\" on host \"%s\" at port \"%s\".\n"),
db, PQuser(pset.db), host, PQport(pset.db));
}
+ printCompressionInfo();
printSSLInfo();
printGSSInfo();
}
@@ -3656,6 +3658,21 @@ connection_warnings(bool in_startup)
}
}
+/*
+ * printCompressionInfo
+ *
+ * Print information about used compressor/decompressor
+ */
+static void
+printCompressionInfo(void)
+{
+ char *algorithms = PQcompression (pset.db);
+
+ if (algorithms != NULL)
+ {
+ printf(_("Compression: %s\n"), algorithms);
+ }
+}
/*
* printSSLInfo
diff --git a/src/common/Makefile b/src/common/Makefile
index 31c0dd366d..185c96063f 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,7 +78,9 @@ OBJS_COMMON = \
unicode_norm.o \
username.o \
wait_error.o \
- wchar.o
+ wchar.o \
+ z_stream.o \
+ zpq_stream.o
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
new file mode 100644
index 0000000000..f8adbfced2
--- /dev/null
+++ b/src/common/z_stream.c
@@ -0,0 +1,669 @@
+#include "c.h"
+#include "pg_config.h"
+#include "common/z_stream.h"
+
+/*
+ * Functions implementing streaming compression algorithm
+ */
+typedef struct
+{
+ /*
+ * Name of compression algorithm.
+ */
+ char const *(*name) (void);
+
+ /*
+ * Create new compression stream. level: compression level
+ */
+ void *(*create_compressor) (int level);
+
+ /*
+ * Create new decompression stream.
+ */
+ void *(*create_decompressor) ();
+
+ /*
+ * Decompress up to "src_size" compressed bytes from *src and write up to
+ * "dst_size" raw (decompressed) bytes to *dst. Number of decompressed
+ * bytes written to *dst is stored in *dst_processed. Number of compressed
+ * bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during decompression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * decompressor internal buffers.
+ *
+ * ZS_STREAM_END if encountered end of compressed data stream.
+ *
+ * ZS_DECOMPRESS_ERROR if encountered an error during decompression
+ * attempt.
+ */
+ ssize_t (*decompress) (void *ds, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Compress up to "src_size" raw (non-compressed) bytes from *src and
+ * write up to "dst_size" compressed bytes to *dst. Number of compressed
+ * bytes written to *dst is stored in *dst_processed. Number of
+ * non-compressed bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during compression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * compressor internal buffers.
+ *
+ * ZS_COMPRESS_ERROR if encountered an error during compression attempt.
+ */
+ ssize_t (*compress) (void *cs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Free compression stream created by create_compressor function.
+ */
+ void (*free_compressor) (void *cs);
+
+ /*
+ * Free decompression stream created by create_decompressor function.
+ */
+ void (*free_decompressor) (void *ds);
+
+ /*
+ * Get compressor error message.
+ */
+ char const *(*compress_error) (void *cs);
+
+ /*
+ * Get decompressor error message.
+ */
+ char const *(*decompress_error) (void *ds);
+
+ ssize_t (*end_compression) (void *cs, void *dst, size_t dst_size, size_t *dst_processed);
+} ZAlgorithm;
+
+struct ZStream
+{
+ ZAlgorithm const *algorithm;
+ void *stream;
+ bool not_flushed;
+};
+
+#if HAVE_LIBZ
+
+#include <stdlib.h>
+#include <zlib.h>
+
+
+static void *
+zlib_create_compressor(int level)
+{
+ int rc;
+ z_stream *c_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(c_stream, 0, sizeof(*c_stream));
+ rc = deflateInit(c_stream, level);
+ if (rc != Z_OK)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+zlib_create_decompressor()
+{
+ int rc;
+ z_stream *d_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(d_stream, 0, sizeof(*d_stream));
+ rc = inflateInit(d_stream);
+ if (rc != Z_OK)
+ {
+ free(d_stream);
+ return NULL;
+ }
+ return d_stream;
+}
+
+static ssize_t
+zlib_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *ds = (z_stream *) d_stream;
+ int rc;
+
+ ds->next_in = (Bytef *) src;
+ ds->avail_in = src_size;
+ ds->next_out = (Bytef *) dst;
+ ds->avail_out = dst_size;
+
+ rc = inflate(ds, Z_SYNC_FLUSH);
+ *src_processed = src_size - ds->avail_in;
+ *dst_processed = dst_size - ds->avail_out;
+
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_STREAM_END;
+ }
+ if (rc != Z_OK && rc != Z_BUF_ERROR)
+ {
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc PG_USED_FOR_ASSERTS_ONLY;
+ unsigned deflate_pending = 0;
+
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = (Bytef *) src;
+ cs->avail_in = src_size;
+
+ rc = deflate(cs, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ *dst_processed = dst_size - cs->avail_out;
+ *src_processed = src_size - cs->avail_in;
+
+ deflatePending(cs, &deflate_pending, Z_NULL); /* check if any data left
+ * in deflate buffer */
+ if (deflate_pending > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+zlib_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = NULL;
+ cs->avail_in = 0;
+
+ rc = deflate(cs, Z_STREAM_END);
+ Assert(rc == Z_OK || rc == Z_STREAM_END);
+ *dst_processed = dst_size - cs->avail_out;
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_OK;
+ }
+
+ return ZS_DATA_PENDING;
+}
+
+static void
+zlib_free_compressor(void *c_stream)
+{
+ z_stream *cs = (z_stream *) c_stream;
+
+ if (cs != NULL)
+ {
+ deflateEnd(cs);
+ free(cs);
+ }
+}
+
+static void
+zlib_free_decompressor(void *d_stream)
+{
+ z_stream *ds = (z_stream *) d_stream;
+
+ if (ds != NULL)
+ {
+ inflateEnd(ds);
+ free(ds);
+ }
+}
+
+static char const *
+zlib_error(void *stream)
+{
+ z_stream *zs = (z_stream *) stream;
+
+ return zs->msg;
+}
+
+static char const *
+zlib_name(void)
+{
+ return "zlib";
+}
+
+#endif
+
+#if USE_LZ4
+#include <lz4.h>
+
+#define MESSAGE_MAX_BYTES 819200
+#define RING_BUFFER_BYTES (1024 * 64 + MESSAGE_MAX_BYTES)
+
+typedef struct ZS_LZ4_CStream
+{
+ LZ4_stream_t *stream;
+ int level;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_CStream;
+
+typedef struct ZS_LZ4_DStream
+{
+ LZ4_streamDecode_t *stream;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_DStream;
+
+static void *
+lz4_create_compressor(int level)
+{
+ ZS_LZ4_CStream *c_stream = (ZS_LZ4_CStream *) malloc(sizeof(ZS_LZ4_CStream));
+
+ if (c_stream == NULL)
+ {
+ return NULL;
+ }
+ c_stream->stream = LZ4_createStream();
+ c_stream->level = level;
+ c_stream->buf_pos = 0;
+ if (c_stream->stream == NULL)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+lz4_create_decompressor()
+{
+ ZS_LZ4_DStream *d_stream = (ZS_LZ4_DStream *) malloc(sizeof(ZS_LZ4_DStream));
+
+ if (d_stream == NULL)
+ {
+ return NULL;
+ }
+
+ d_stream->stream = LZ4_createStreamDecode();
+ d_stream->buf_pos = 0;
+ if (d_stream->stream == NULL)
+ {
+ free(d_stream);
+ return NULL;
+ }
+
+ return d_stream;
+}
+
+static ssize_t
+lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+
+ char *const decPtr = &ds->buf[ds->buf_pos];
+
+ const int decBytes = LZ4_decompress_safe_continue(
+ ds->stream, src, decPtr, (int) src_size, (int) dst_size);
+
+ Assert(decBytes > 0);
+
+
+ *dst_processed = decBytes;
+ *src_processed = src_size;
+
+ memcpy(dst, decPtr, decBytes); /* write msg length */
+
+ ds->buf_pos += decBytes;
+ if (ds->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ ds->buf_pos = 0;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+ int cmpBytes;
+
+
+
+ src_size = Min(MESSAGE_MAX_BYTES, src_size);
+
+ memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
+
+ Assert(dst_size >= LZ4_compressBound(src_size));
+
+ cmpBytes = LZ4_compress_fast_continue(
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, cs->level);
+
+
+ Assert(cmpBytes > 0);
+ Assert(cmpBytes <= MESSAGE_MAX_BYTES);
+
+ *dst_processed = cmpBytes;
+ *src_processed = src_size;
+
+ cs->buf_pos += src_size;
+ if (cs->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ cs->buf_pos = 0;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+lz4_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ *dst_processed = 0;
+ return ZS_OK;
+}
+
+static void
+lz4_free_compressor(void *c_stream)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ if (cs->stream != NULL)
+ {
+ LZ4_freeStream(cs->stream);
+ }
+ free(cs);
+ }
+}
+
+static void
+lz4_free_decompressor(void *d_stream)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ if (ds->stream != NULL)
+ {
+ LZ4_freeStreamDecode(ds->stream);
+ }
+ free(ds);
+ }
+}
+
+static char const *
+lz4_error(void *stream)
+{
+ /* lz4 doesn't have any explicit API to get the error names */
+ return "NO_MSG";
+}
+
+static char const *
+lz4_name(void)
+{
+ return "lz4";
+}
+
+#endif
+
+static char const *
+no_compression_name(void)
+{
+ return NULL;
+}
+
+/*
+ * Array with all supported compression algorithms.
+ */
+static ZAlgorithm const zs_algorithms[] =
+{
+#if HAVE_LIBZ
+ {zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
+#endif
+#if USE_LZ4
+ {lz4_name, lz4_create_compressor, lz4_create_decompressor, lz4_decompress, lz4_compress, lz4_free_compressor, lz4_free_decompressor, lz4_error, lz4_error, lz4_end},
+#endif
+ {no_compression_name}
+};
+
+inline bool
+zs_is_valid_impl_id(unsigned int id)
+{
+ return id >= 0 && id < lengthof(zs_algorithms);
+}
+
+static ssize_t
+zs_init_compressor(ZStream * zs, unsigned int c_alg_impl, int c_level)
+{
+ if (!zs_is_valid_impl_id(c_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[c_alg_impl];
+ zs->stream = zs->algorithm->create_compressor(c_level);
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+static ssize_t
+zs_init_decompressor(ZStream * zs, unsigned int d_alg_impl)
+{
+ if (!zs_is_valid_impl_id(d_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[d_alg_impl];
+ zs->stream = zs->algorithm->create_decompressor();
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Index of used compression algorithm in zs_algorithms array.
+ */
+ZStream *
+zs_create_compressor(unsigned int c_alg_impl, int c_level)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_compressor(zs, c_alg_impl, c_level))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ZStream *
+zs_create_decompressor(unsigned int d_alg_impl)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_decompressor(zs, d_alg_impl))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ssize_t
+zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *src_processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->decompress(zs->stream,
+ src, src_size, src_processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+
+ if (rc == ZS_OK || rc == ZS_INCOMPLETE_SRC || rc == ZS_STREAM_END)
+ {
+ return rc;
+ }
+
+ return ZS_DECOMPRESS_ERROR;
+}
+
+ssize_t
+zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->compress(zs->stream,
+ buf, size, processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+void
+zs_compressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_compressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+void
+zs_decompressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_decompressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+ssize_t
+zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *dst_processed = 0;
+
+ rc = zs->algorithm->end_compression(zs->stream, dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+char const *
+zs_compress_error(ZStream * zs)
+{
+ return zs->algorithm->compress_error(zs->stream);
+}
+
+char const *
+zs_decompress_error(ZStream * zs)
+{
+ return zs->algorithm->decompress_error(zs->stream);
+}
+
+bool
+zs_buffered(ZStream * zs)
+{
+ return zs ? zs->not_flushed : 0;
+}
+
+
+/*
+ * Get list of the supported algorithms.
+ */
+char **
+zs_get_supported_algorithms(void)
+{
+ size_t n_algorithms = lengthof(zs_algorithms);
+ char **algorithm_names = malloc(n_algorithms * sizeof(char *));
+
+ for (size_t i = 0; i < n_algorithms; i++)
+ {
+ algorithm_names[i] = (char *) zs_algorithms[i].name();
+ }
+
+ return algorithm_names;
+}
+
+char const *
+zs_compress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
+
+char const *
+zs_decompress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 0000000000..a712200e45
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,1022 @@
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+#include <unistd.h>
+#include <math.h>
+
+#include "common/zpq_stream.h"
+#include "pg_config.h"
+#include "port/pg_bswap.h"
+
+/* log warnings on backend */
+#ifndef FRONTEND
+#define pg_log_warning(...) elog(WARNING, __VA_ARGS__)
+#else
+#define pg_log_warning(...) (void)0
+#endif
+
+/* ZpqBuffer size, in bytes */
+#define ZPQ_BUFFER_SIZE 8192000
+/* CompressedData msg_type */
+#define ZPQ_COMPRESSED_MSG_TYPE 'm'
+/* SetCompressionMethod msg_type */
+#define ZPQ_SET_COMPRESSION_MSG_TYPE 'k'
+
+#define ZPQ_COMPRESS_THRESHOLD 60
+
+typedef struct ZpqBuffer ZpqBuffer;
+
+
+/* ZpqBuffer used as RX/TX buffer in ZpqStream */
+struct ZpqBuffer
+{
+ char buf[ZPQ_BUFFER_SIZE];
+ size_t size; /* current size of buf */
+ size_t pos; /* current position in buf, in range [0, size] */
+};
+
+static inline void
+zpq_buf_init(ZpqBuffer * zb)
+{
+ zb->size = 0;
+ zb->pos = 0;
+}
+
+static inline size_t
+zpq_buf_left(ZpqBuffer * zb)
+{
+ Assert(zb->buf);
+ return ZPQ_BUFFER_SIZE - zb->size;
+}
+
+static inline size_t
+zpq_buf_unread(ZpqBuffer * zb)
+{
+ return zb->size - zb->pos;
+}
+
+static inline char *
+zpq_buf_size(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->size;
+}
+
+static inline char *
+zpq_buf_pos(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->pos;
+}
+
+static inline void
+zpq_buf_size_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->size += value;
+}
+
+static inline void
+zpq_buf_pos_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->pos += value;
+}
+
+static inline void
+zpq_buf_reuse(ZpqBuffer * zb)
+{
+ size_t unread = zpq_buf_unread(zb);
+
+ if (unread > 5) /* can read message header, don't do anything */
+ return;
+ if (unread == 0)
+ {
+ zb->size = 0;
+ zb->pos = 0;
+ return;
+ }
+ memmove(zb->buf, zb->buf + zb->pos, unread);
+ zb->size = unread;
+ zb->pos = 0;
+}
+
+struct ZpqStream
+{
+ ZStream *c_stream; /* underlying compression stream */
+ ZStream *d_stream; /* underlying decompression stream */
+
+ size_t tx_total; /* amount of bytes sent to tx_func */
+
+ size_t tx_total_raw; /* amount of bytes received by zpq_write */
+ size_t rx_total; /* amount of bytes read by rx_func */
+ size_t rx_total_raw; /* amount of bytes returned by zpq_write */
+ bool is_compressing; /* current compression state */
+
+ bool is_decompressing; /* current decompression state */
+ size_t rx_msg_bytes_left; /* number of bytes left to process without
+ * changing the decompression state */
+ size_t tx_msg_bytes_left; /* number of bytes left to process without
+ * changing the compression state */
+
+ ZpqBuffer rx_in; /* buffer for unprocessed data read by rx_func */
+ ZpqBuffer tx_in; /* buffer for unprocessed data consumed by
+ * zpq_write */
+ ZpqBuffer tx_out; /* buffer for processed data waiting for send
+ * via tx_func */
+
+ zpq_rx_func rx_func;
+ zpq_tx_func tx_func;
+ void *arg;
+
+ zpq_compressor *compressors; /* compressors array holds the available
+ * compressors to use for
+ * compression/decompression */
+ size_t n_compressors; /* size of the compressors array */
+ int compress_alg_idx; /* index of the active compression
+ * algorithm */
+ int decompress_alg_idx; /* index of the active decompression
+ * algorithm */
+ int compressor_by_msg_type[256]; /* map to choose a compressor
+ * by the protocol message
+ * type */
+
+ bool reading_set_compression; /* utility marker indicating
+ * partial SetCompressionMethod
+ * read */
+};
+
+/*
+ * Message compression map defines the logic for choosing the compressor
+ * based on the protocol message type. Currently, it is a basic prototype to demonstrate
+ * the capabilities of the on-the-fly compression switch.
+ */
+static inline void
+zpq_build_msg_compression_map(ZpqStream * zpq)
+{
+ int i;
+
+ for (i = 0; i < 256; i++)
+ {
+ zpq->compressor_by_msg_type[i] = -1;
+ }
+
+ for (i = 0; i < zpq->n_compressors; i++)
+ {
+ /* compress CopyData, DataRow and Query messages */
+ if (zpq->compressor_by_msg_type['d'] == -1)
+ {
+ zpq->compressor_by_msg_type['d'] = i;
+ }
+ if (zpq->compressor_by_msg_type['D'] == -1)
+ {
+ zpq->compressor_by_msg_type['D'] = i;
+ }
+ if (zpq->compressor_by_msg_type['Q'] == -1)
+ {
+ zpq->compressor_by_msg_type['Q'] = i;
+ }
+ }
+}
+
+/*
+ * Choose the index of compressor to use for the message of msg_type with msg_len.
+ * Return values:
+ * - the non-negative index of zpq->compressors array
+ * - -1, if message should not be compressed
+ */
+static inline int
+zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
+ {
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
+ }
+ return -1;
+}
+
+/*
+ * Check if should compress message of msg_type with msg_len.
+ * Return true if should, false if should not.
+ */
+static inline bool
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
+}
+
+/*
+ * Check if message is a CompressedData.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_compressed_msg(char msg_type)
+{
+ return msg_type == ZPQ_COMPRESSED_MSG_TYPE;
+}
+
+/*
+ * Check if message is a SetCompressionMethod.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_set_compression_msg(char msg_type)
+{
+ return msg_type == ZPQ_SET_COMPRESSION_MSG_TYPE;
+}
+
+ZpqStream *
+zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size)
+{
+ ZpqStream *zpq;
+
+ /* zpqStream needs at least one compressor */
+ if (n_compressors == 0 || compressors == NULL)
+ {
+ return NULL;
+ }
+ zpq = (ZpqStream *) malloc(sizeof(ZpqStream));
+
+ zpq->compressors = compressors;
+ zpq->n_compressors = n_compressors;
+ zpq->compress_alg_idx = -1;
+ zpq->decompress_alg_idx = -1;
+
+ zpq->is_compressing = false;
+ zpq->is_decompressing = false;
+ zpq->rx_msg_bytes_left = 0;
+ zpq->tx_msg_bytes_left = 0;
+ zpq_buf_init(&zpq->tx_in);
+
+ zpq->tx_total = 0;
+ zpq->tx_total_raw = 0;
+ zpq->rx_total = 0;
+ zpq->rx_total_raw = 0;
+
+ zpq_buf_init(&zpq->rx_in);
+ zpq_buf_size_advance(&zpq->rx_in, rx_data_size);
+ Assert(rx_data_size < ZPQ_BUFFER_SIZE);
+ memcpy(zpq->rx_in.buf, rx_data, rx_data_size);
+
+ zpq_buf_init(&zpq->tx_out);
+
+ zpq->rx_func = rx_func;
+ zpq->tx_func = tx_func;
+ zpq->arg = arg;
+ zpq->reading_set_compression = false;
+ zpq->c_stream = NULL;
+ zpq->d_stream = NULL;
+
+ zpq_build_msg_compression_map(zpq);
+
+ return zpq;
+}
+
+/* Compress up to src_size bytes from *src into CompressedData and write it to the tx buffer.
+ * Returns ZS_OK on success, ZS_COMPRESS_ERROR if encountered a compression error. */
+static inline ssize_t
+zpq_write_compressed_message(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ size_t compressed_len;
+ ssize_t rc;
+ uint32 size;
+
+ /* check if have enough space */
+ if (zpq_buf_left(&zpq->tx_out) <= 5)
+ {
+ /* too little space for CompressedData, abort */
+ *src_processed = 0;
+ return ZS_OK;
+ }
+
+ compressed_len = 0;
+ rc = zs_write(zpq->c_stream, src, src_size, src_processed,
+ zpq_buf_size(&zpq->tx_out) + 5, zpq_buf_left(&zpq->tx_out) - 5, &compressed_len);
+
+ if (compressed_len > 0)
+ {
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_COMPRESSED_MSG_TYPE;
+ size = pg_hton32(compressed_len + 4);
+
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &size, sizeof(uint32)); /* write msg length */
+ compressed_len += 5; /* append header length to compressed data
+ * length */
+ }
+
+ zpq->tx_total_raw += *src_processed;
+ zpq->tx_total += compressed_len;
+ zpq_buf_size_advance(&zpq->tx_out, compressed_len);
+ return rc;
+}
+
+/* Copy the data directly from *src to the tx buffer */
+static void
+zpq_write_uncompressed(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ src_size = Min(zpq_buf_left(&zpq->tx_out), src_size);
+ memcpy(zpq_buf_size(&zpq->tx_out), src, src_size);
+
+ zpq->tx_total_raw += src_size;
+ zpq->tx_total += src_size;
+ zpq_buf_size_advance(&zpq->tx_out, src_size);
+ *src_processed = src_size;
+}
+
+static ssize_t
+zpq_write_set_compression_msg(ZpqStream * zpq, int new_compress_idx)
+{
+ uint32 len;
+ uint8 idx;
+
+ /*
+ * check if have enough space: msg_type(1 byte) + msg_len(4 bytes) +
+ * compress_alg_idx(1 byte)
+ */
+ if (zpq_buf_left(&zpq->tx_out) < 6)
+ {
+ return -1;
+ }
+
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_SET_COMPRESSION_MSG_TYPE;
+ len = pg_hton32(5);
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &len, sizeof(uint32)); /* write msg length */
+
+ /* currently we expect idx to be in range [0, 255] */
+ Assert(new_compress_idx >= 0 && new_compress_idx <= UINT8_MAX);
+ idx = (uint8) new_compress_idx;
+ memcpy(zpq_buf_size(&zpq->tx_out) + 5, &idx, sizeof(uint8)); /* write
+ * new_compress_idx */
+
+ zpq->tx_total_raw += 6;
+ zpq->tx_total += 6;
+ zpq_buf_size_advance(&zpq->tx_out, 6);
+ return 0;
+}
+
+/* Determine if should compress the next message and change the current compression state */
+static ssize_t
+zpq_toggle_compression(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ int new_compress_idx = zpq_choose_compressor(zpq, msg_type, msg_len);
+ bool should_compress = new_compress_idx != -1;
+
+ /*
+ * negative new_compress_idx indicates that we should not compress this
+ * message
+ */
+ if (should_compress)
+ {
+ /*
+ * if the new compressor does not match the current one, process the
+ * switch
+ */
+ if (zpq->compress_alg_idx != new_compress_idx)
+ {
+ if (zpq_write_set_compression_msg(zpq, new_compress_idx))
+ {
+ /*
+ * come back later when we can write the entire
+ * SetCompressionMethod message
+ */
+ return 0;
+ }
+
+ zs_compressor_free(zpq->c_stream);
+ zpq->c_stream = zs_create_compressor(zpq->compressors[new_compress_idx].impl, zpq->compressors[new_compress_idx].level);
+ if (zpq->c_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->compress_alg_idx = new_compress_idx;
+ }
+ }
+
+ zpq->is_compressing = should_compress;
+ zpq->tx_msg_bytes_left = msg_len + 1;
+ return 0;
+}
+
+/*
+ * Internal write function. Reads the data from *src buffer,
+ * determines the postgres messages type and length.
+ * If message matches the compression criteria, it wraps the message into
+ * CompressedData. Otherwise, leaves the message unchanged.
+ * If *src data ends with incomplete message header, this function is not
+ * going to read this message header.
+ * Returns number of written raw bytes or error code.
+ * In the last case number of bytes written is stored in *processed.
+ */
+static ssize_t
+zpq_write_internal(ZpqStream * zpq, void const *src, size_t src_size, size_t *processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ do
+ {
+ /*
+ * try to read ahead the next message types and increase
+ * tx_msg_bytes_left, if possible
+ */
+ while (zpq->tx_msg_bytes_left > 0 && src_size - src_pos >= zpq->tx_msg_bytes_left + 5)
+ {
+ char msg_type = *((char *) src + src_pos + zpq->tx_msg_bytes_left);
+ uint32 msg_len;
+
+ memcpy(&msg_len, (char *) src + src_pos + zpq->tx_msg_bytes_left + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ if (zpq_should_compress(zpq, msg_type, msg_len) != zpq->is_compressing)
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+ zpq->tx_msg_bytes_left += msg_len + 1;
+ }
+
+ /*
+ * Write CompressedData if currently is compressing or have some
+ * buffered data left in underlying compression stream
+ */
+ if (zs_buffered(zpq->c_stream) || (zpq->is_compressing && zpq->tx_msg_bytes_left > 0))
+ {
+ size_t buf_processed = 0;
+ size_t to_compress = Min(zpq->tx_msg_bytes_left, src_size - src_pos);
+
+ rc = zpq_write_compressed_message(zpq, (char *) src + src_pos, to_compress, &buf_processed);
+ src_pos += buf_processed;
+ zpq->tx_msg_bytes_left -= buf_processed;
+
+ if (rc != ZS_OK)
+ {
+ *processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * If not going to compress the data from *src, just write it
+ * uncompressed.
+ */
+ else if (zpq->tx_msg_bytes_left > 0)
+ { /* determine next message type */
+ size_t copy_len = Min(src_size - src_pos, zpq->tx_msg_bytes_left);
+ size_t copy_processed = 0;
+
+ zpq_write_uncompressed(zpq, (char *) src + src_pos, copy_len, ©_processed);
+ src_pos += copy_processed;
+ zpq->tx_msg_bytes_left -= copy_processed;
+ }
+
+ /*
+ * Reached the compression toggle point, fetch next message header to
+ * determine compression state.
+ */
+ else
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ if (src_size - src_pos < 5)
+ {
+ /*
+ * must return here because we can't continue without full
+ * message header
+ */
+ *processed = src_pos;
+ return ZPQ_INCOMPLETE_HEADER;
+ }
+
+ msg_type = *((char *) src + src_pos);
+ memcpy(&msg_len, (char *) src + src_pos + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ rc = zpq_toggle_compression(zpq, msg_type, msg_len);
+ if (rc)
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * repeat sending while there is some data in input or internal
+ * compression buffer
+ */
+ } while (src_pos < src_size && zpq_buf_left(&zpq->tx_out) > 6);
+
+ return src_pos;
+}
+
+ssize_t
+zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ /* try to process as much data as possible before calling the tx_func */
+ while (zpq_buf_left(&zpq->tx_out) > 6)
+ {
+ size_t copy_len = Min(zpq_buf_left(&zpq->tx_in), src_size - src_pos);
+ size_t processed;
+
+ memcpy(zpq_buf_size(&zpq->tx_in), (char *) src + src_pos, copy_len);
+ zpq_buf_size_advance(&zpq->tx_in, copy_len);
+ src_pos += copy_len;
+
+ if (zpq_buf_unread(&zpq->tx_in) == 0 && !zs_buffered(zpq->c_stream))
+ {
+ break;
+ }
+
+ processed = 0;
+
+ rc = zpq_write_internal(zpq, zpq_buf_pos(&zpq->tx_in), zpq_buf_unread(&zpq->tx_in), &processed);
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, rc);
+ zpq_buf_reuse(&zpq->tx_in);
+ }
+ else
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, processed);
+ zpq_buf_reuse(&zpq->tx_in);
+ if (rc == ZPQ_INCOMPLETE_HEADER)
+ {
+ break;
+ }
+ *src_processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * call the tx_func if have any bytes to send
+ */
+ while (zpq_buf_unread(&zpq->tx_out))
+ {
+ rc = zpq->tx_func(zpq->arg, zpq_buf_pos(&zpq->tx_out), zpq_buf_unread(&zpq->tx_out));
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_out, rc);
+ }
+ else
+ {
+ *src_processed = src_pos;
+ zpq_buf_reuse(&zpq->tx_out);
+ return rc;
+ }
+ }
+
+ zpq_buf_reuse(&zpq->tx_out);
+ return src_pos;
+}
+
+/* Decompress bytes from RX buffer and write up to dst_len of uncompressed data to *dst.
+ * Returns:
+ * ZS_OK on success,
+ * ZS_STREAM_END if reached end of compressed chunk
+ * ZS_DECOMPRESS_ERROR if encountered a decompression error */
+static inline ssize_t
+zpq_read_compressed_message(ZpqStream * zpq, char *dst, size_t dst_len, size_t *dst_processed)
+{
+ size_t rx_processed = 0;
+ ssize_t rc;
+ size_t read_len = Min(zpq->rx_msg_bytes_left, zpq_buf_unread(&zpq->rx_in));
+
+ Assert(read_len == zpq->rx_msg_bytes_left);
+ rc = zs_read(zpq->d_stream, zpq_buf_pos(&zpq->rx_in), read_len, &rx_processed,
+ dst, dst_len, dst_processed);
+
+ zpq_buf_pos_advance(&zpq->rx_in, rx_processed);
+ zpq->rx_total_raw += *dst_processed;
+ zpq->rx_msg_bytes_left -= rx_processed;
+ return rc;
+}
+
+/* Copy up to dst_len bytes from rx buffer to *dst.
+ * Returns amount of bytes copied. */
+static inline size_t
+zpq_read_uncompressed(ZpqStream * zpq, char *dst, size_t dst_len)
+{
+ size_t copy_len;
+
+ Assert(zpq_buf_unread(&zpq->rx_in) > 0);
+ copy_len = Min(zpq->rx_msg_bytes_left, Min(zpq_buf_unread(&zpq->rx_in), dst_len));
+
+ memcpy(dst, zpq_buf_pos(&zpq->rx_in), copy_len);
+
+ zpq_buf_pos_advance(&zpq->rx_in, copy_len);
+ zpq->rx_total_raw += copy_len;
+ zpq->rx_msg_bytes_left -= copy_len;
+ return copy_len;
+}
+
+/* Determine if should decompress the next message and
+ * change the current decompression state */
+static inline void
+zpq_toggle_decompression(ZpqStream * zpq)
+{
+ uint32 msg_len;
+ char msg_type = *zpq_buf_pos(&zpq->rx_in);
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+
+ if (zpq_is_set_compression_msg(msg_type))
+ {
+ Assert(msg_len == 5);
+ zpq->reading_set_compression = true;
+ /* set compression message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ }
+ else
+ {
+ zpq->is_decompressing = zpq_is_compressed_msg(msg_type);
+ zpq->rx_msg_bytes_left = msg_len + 1;
+
+ if (zpq->is_decompressing)
+ {
+ /* compressed message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ zpq->rx_msg_bytes_left -= 5;
+ }
+ }
+}
+
+static inline ssize_t
+zpq_process_switch(ZpqStream * zpq)
+{
+ uint8 algorithm_idx;
+
+ if (zpq_buf_unread(&zpq->rx_in) < 1)
+ {
+ return 0;
+ }
+
+ algorithm_idx = *zpq_buf_pos(&zpq->rx_in);
+
+ zpq_buf_pos_advance(&zpq->rx_in, 1);
+ zpq->reading_set_compression = false;
+
+ if (algorithm_idx != zpq->decompress_alg_idx)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ zpq->d_stream = zs_create_decompressor(zpq->compressors[algorithm_idx].impl);
+ if (zpq->d_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->decompress_alg_idx = algorithm_idx;
+ }
+
+ return 0;
+}
+
+ssize_t
+zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock)
+{
+ size_t dst_pos = 0;
+ size_t dst_processed = 0;
+ ssize_t rc;
+
+ /* Read until some data fetched */
+ while (dst_pos == 0)
+ {
+ zpq_buf_reuse(&zpq->rx_in);
+
+ if (!zpq_buffered_rx(zpq) || (zpq->is_decompressing && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left))
+ {
+ if (noblock)
+ {
+ /*
+ * can't read anything w/o the potentially blocking backend
+ * call
+ */
+ return dst_pos;
+ }
+ rc = zpq->rx_func(zpq->arg, zpq_buf_size(&zpq->rx_in), zpq_buf_left(&zpq->rx_in));
+ if (rc > 0) /* read fetches some data */
+ {
+ zpq->rx_total += rc;
+ zpq_buf_size_advance(&zpq->rx_in, rc);
+ }
+ else /* read failed */
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * try to read ahead the next message types and increase
+ * rx_msg_bytes_left, if possible (ONLY UNCOMPRESSED MESSAGES)
+ */
+ while (!zpq->is_decompressing && zpq->rx_msg_bytes_left > 0 && (zpq_buf_unread(&zpq->rx_in) >= zpq->rx_msg_bytes_left + 5))
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ msg_type = *(zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left);
+ if (zpq_is_compressed_msg(msg_type) || zpq_is_set_compression_msg(msg_type))
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left + 1, 4);
+ zpq->rx_msg_bytes_left += pg_ntoh32(msg_len) + 1;
+ }
+
+
+ if (zpq->rx_msg_bytes_left > 0 || zs_buffered(zpq->d_stream))
+ {
+ dst_processed = 0;
+ if (zpq->is_decompressing || zs_buffered(zpq->d_stream))
+ {
+ if (!zs_buffered(zpq->d_stream) && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left)
+ {
+ /*
+ * prefer to read only the fully compressed messages or
+ * read if some data is buffered
+ */
+ continue;
+ }
+ rc = zpq_read_compressed_message(zpq, dst, dst_size - dst_pos, &dst_processed);
+ dst_pos += dst_processed;
+ if (rc == ZS_STREAM_END)
+ {
+ continue;
+ }
+ if (rc != ZS_OK)
+ {
+ return rc;
+ }
+ }
+ else
+ dst_pos += zpq_read_uncompressed(zpq, dst, dst_size - dst_pos);
+ }
+ else if (zpq->reading_set_compression)
+ {
+ zpq_process_switch(zpq);
+ }
+ else if (zpq_buf_unread(&zpq->rx_in) >= 5)
+ zpq_toggle_decompression(zpq);
+ }
+ return dst_pos;
+}
+
+bool
+zpq_buffered_rx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->rx_in) >= 5 || (zpq_buf_unread(&zpq->rx_in) > 0 && zpq->rx_msg_bytes_left > 0) ||
+ zs_buffered(zpq->d_stream) : 0;
+}
+
+bool
+zpq_buffered_tx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->tx_in) >= 5 || (zpq_buf_unread(&zpq->tx_in) > 0 && zpq->tx_msg_bytes_left > 0) || zpq_buf_unread(&zpq->tx_out) > 0 ||
+ zs_buffered(zpq->c_stream) : 0;
+}
+
+void
+zpq_free(ZpqStream * zpq)
+{
+ if (zpq)
+ {
+ if (zpq->c_stream)
+ {
+ zs_compressor_free(zpq->c_stream);
+ }
+ if (zpq->d_stream)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ }
+ free(zpq);
+ }
+}
+
+char const *
+zpq_compress_error(ZpqStream * zpq)
+{
+ return zs_compress_error(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_error(ZpqStream * zpq)
+{
+ return zs_decompress_error(zpq->d_stream);
+}
+
+char const *
+zpq_compress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_compress_algorithm_name(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_decompress_algorithm_name(zpq->d_stream);
+}
+
+char *
+zpq_algorithms(ZpqStream * zpq)
+{
+ return zpq_serialize_compressors(zpq->compressors, zpq->n_compressors);
+}
+
+int
+zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int i;
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *protocol_extension = strchr(val, ';');
+
+ *compressors = NULL;
+ *n_compressors = 0;
+
+ /* No protocol extensions are currently supported */
+ if (protocol_extension)
+ *protocol_extension = '\0';
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ if (pg_strcasecmp(val, "true") == 0 ||
+ pg_strcasecmp(val, "yes") == 0 ||
+ pg_strcasecmp(val, "on") == 0 ||
+ pg_strcasecmp(val, "any") == 0 ||
+ pg_strcasecmp(val, "1") == 0)
+ {
+ /* return all available compressors */
+ *n_compressors = n_supported_algorithms;
+
+ if (n_supported_algorithms)
+ {
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+ for (i = 0; i < n_supported_algorithms; i++)
+ {
+ (*compressors)[i].impl = i;
+ (*compressors)[i].level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ }
+ }
+ return 1;
+ }
+
+ if (*val == 0 ||
+ pg_strcasecmp(val, "false") == 0 ||
+ pg_strcasecmp(val, "no") == 0 ||
+ pg_strcasecmp(val, "off") == 0 ||
+ pg_strcasecmp(val, "0") == 0)
+ {
+ /* Compression is disabled */
+ return 0;
+ }
+
+ return zpq_deserialize_compressors(val, compressors, n_compressors) ? 1 : -1;
+}
+
+bool
+zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int selected_alg_mask = 0; /* bitmask of already selected
+ * algorithms to avoid duplicates in
+ * compressors */
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *c_string_dup = strdup(c_string); /* following parsing can
+ * modify the string */
+ char *p = c_string_dup;
+
+ *n_compressors = 0;
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+
+ while (*p != '\0')
+ {
+ char *sep = strchr(p, ',');
+ char *col;
+ int compression_level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ bool found;
+
+ if (sep != NULL)
+ *sep = '\0';
+
+ col = strchr(p, ':');
+ if (col != NULL)
+ {
+ *col = '\0';
+ if (sscanf(col + 1, "%d", &compression_level) != 1)
+ {
+ pg_log_warning("invalid compression level %s in compression option '%s'", col + 1, p);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+ }
+ found = false;
+ for (int i = 0; supported_algorithms[i] != NULL; i++)
+ {
+ if (pg_strcasecmp(p, supported_algorithms[i]) == 0)
+ {
+ if (selected_alg_mask & (1 << i))
+ {
+ /* duplicates are not allowed */
+ pg_log_warning("duplicate algorithm %s in compressors string %s", p, c_string);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+
+ (*compressors)[*n_compressors].impl = i;
+ (*compressors)[*n_compressors].level = compression_level;
+
+ selected_alg_mask |= 1 << i;
+ *n_compressors += 1;
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ pg_log_warning("algorithm %s is not supported", p);
+ }
+ if (sep)
+ p = sep + 1;
+ else
+ break;
+ }
+
+ if (*n_compressors == 0)
+ {
+ free(*compressors);
+ *compressors = NULL;
+ }
+ free(c_string_dup);
+ return true;
+}
+
+char *
+zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors)
+{
+ char *res;
+ char *p;
+ size_t i;
+ size_t total_len = 0;
+ char **supported_algorithms = zs_get_supported_algorithms();
+
+ if (n_compressors == 0)
+ {
+ return NULL;
+ }
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ size_t level_len;
+
+ if (!zs_is_valid_impl_id(compressors[i].impl))
+ {
+ pg_log_warning("algorithm impl_id %d is incorrect", compressors[i].impl);
+ return NULL;
+ }
+
+ /* determine the length of the compression level string */
+ level_len = compressors[i].level == 0 ? 1 : (int) floor(log10(abs(compressors[i].level))) + 1;
+ if (compressors[i].level < 0)
+ {
+ level_len += 1; /* add the leading "-" */
+ }
+
+ /*
+ * single entry looks like "alg_name:compression_level," so +2 is for
+ * ":" and "," symbols (or trailing null)
+ */
+ total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
+ }
+
+ res = p = malloc(total_len);
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
+ if (i < n_compressors - 1)
+ *p++ = ',';
+ }
+ return res;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b6f689e8d1..04d185369c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5334,9 +5334,9 @@
proname => 'pg_stat_get_activity', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'int4',
- proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8}',
- proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id}',
+ proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
prosrc => 'pg_stat_get_activity' },
{ oid => '3318',
descr => 'statistics: information about progress of backends running maintenance command',
@@ -5644,6 +5644,14 @@
proargnames => '{wal_records,wal_fpi,wal_bytes,wal_buffers_full,wal_write,wal_sync,wal_write_time,wal_sync_time,stats_reset}',
prosrc => 'pg_stat_get_wal' },
+{ oid => '9598', descr => 'statistics: information about network traffic',
+ proname => 'pg_stat_get_network_traffic', proisstrict => 'f', provolatile => 's',
+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o}',
+ proargnames => '{_beid,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
+ prosrc => 'pg_stat_get_network_traffic' },
+
{ oid => '2306', descr => 'statistics: information about SLRU caches',
proname => 'pg_stat_get_slru', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
@@ -5785,6 +5793,10 @@
proname => 'pg_tablespace_location', provolatile => 's', prorettype => 'text',
proargtypes => 'oid', prosrc => 'pg_tablespace_location' },
+{ oid => '9257', descr => 'connection compression algorithm',
+ proname => 'pg_compression_algorithm', provolatile => 's', prorettype => 'text',
+ proargtypes => '', prosrc => 'pg_compression_algorithm' },
+
{ oid => '1946',
descr => 'convert bytea value into some ascii-only text string',
proname => 'encode', prorettype => 'text', proargtypes => 'bytea text',
diff --git a/src/include/common/z_stream.h b/src/include/common/z_stream.h
new file mode 100644
index 0000000000..4316563aa7
--- /dev/null
+++ b/src/include/common/z_stream.h
@@ -0,0 +1,109 @@
+/*
+ * z_stream.h
+ * Streaming compression
+ */
+
+
+#ifndef Z_STREAM_H
+#define Z_STREAM_H
+
+#include <stdlib.h>
+
+#define ZS_OK (0)
+#define ZS_IO_ERROR (-1)
+#define ZS_DECOMPRESS_ERROR (-2)
+#define ZS_COMPRESS_ERROR (-3)
+#define ZS_STREAM_END (-4)
+#define ZS_DATA_PENDING (-5)
+#define ZS_INCOMPLETE_SRC (-6) /* cannot decompress unless full src message
+ * is fetched */
+#define ZS_EXPAND_DST (-7) /* cannot compress: dst buffer is too small
+ * for given src size */
+
+struct ZStream;
+typedef struct ZStream ZStream;
+
+#endif
+
+/*
+ * Create compression stream for sending compressed data.
+ * c_alg_impl: index of chosen compression algorithm
+ * c_level: compression c_level
+ */
+extern ZStream * zs_create_compressor(unsigned int c_alg_impl, int c_level);
+
+/*
+ * Create decompression stream for reading compressed data.
+ * d_alg_impl: index of chosen decompression algorithm
+ */
+extern ZStream * zs_create_decompressor(unsigned int d_alg_impl);
+
+/*
+ * Read up to "size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZS_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Write up to "size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZS_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *processed.
+ */
+extern ssize_t zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zs_decompress_error(ZStream * zs);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zs_compress_error(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal rx decompression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal tx compression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * End the compression stream.
+ */
+extern ssize_t zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Free stream created by zs_create_compressor function.
+ */
+extern void zs_compressor_free(ZStream * zs);
+
+/*
+ * Free stream created by zs_create_decompressor function.
+ */
+extern void zs_decompressor_free(ZStream * zs);
+
+/*
+ * Get the name of chosen compression algorithm.
+ */
+extern char const *zs_compress_algorithm_name(ZStream * zs);
+
+/*
+ * Get the name of chosen decompression algorithm.
+ */
+extern char const *zs_decompress_algorithm_name(ZStream * zs);
+
+/*
+ Returns zero terminated array with compression algorithms names
+*/
+extern char **zs_get_supported_algorithms(void);
+
+/*
+ Returns true if provided id is a valid compression algorithm id, otherwise returns false
+*/
+extern bool zs_is_valid_impl_id(unsigned int id);
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 0000000000..29b467015f
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,120 @@
+/*
+ * zpq_stream.h
+ * Streaming compression for libpq
+ */
+#include "z_stream.h"
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#define ZPQ_DEFAULT_COMPRESSION_LEVEL (1)
+#define ZPQ_INCOMPLETE_HEADER (-6)
+#define ZPQ_FATAL_ERROR (-7)
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t (*zpq_tx_func) (void *arg, void const *data, size_t size);
+typedef ssize_t (*zpq_rx_func) (void *arg, void *data, size_t size);
+
+/*
+ * Descriptor of compression algorithm chosen by client
+ */
+typedef struct zpq_compressor
+{
+ unsigned int impl; /* compression algorithm index */
+ int level; /* compression level */
+} zpq_compressor;
+
+#endif
+
+/*
+ * Create compression stream with rx/tx function for reading/sending compressed data.
+ * tx_func: function for writing compressed data in underlying stream
+ * rx_func: function for reading compressed data from underlying stream
+ * arg: context passed to the function
+ * rx_data: received data (compressed data already fetched from input stream)
+ * rx_data_size: size of data fetched from input stream
+ */
+extern ZpqStream * zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size);
+
+/*
+ * Write up to "src_size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZPQ_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *src_processed.
+ */
+extern ssize_t zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed);
+
+/*
+ * Read up to "dst_size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZPQ_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock);
+
+/*
+ * Return true if non-flushed data left in internal rx decompression buffer.
+ */
+extern bool zpq_buffered_rx(ZpqStream * zpq);
+
+/*
+ * Return true if non-flushed data left in internal tx compression buffer.
+ */
+extern bool zpq_buffered_tx(ZpqStream * zpq);
+
+/*
+ * Free stream created by zs_create function.
+ */
+extern void zpq_free(ZpqStream * zpq);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zpq_decompress_error(ZpqStream * zpq);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zpq_compress_error(ZpqStream * zpq);
+
+/*
+ * Get the name of the current compression algorithm.
+ */
+extern char const *zpq_compress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Get the name of the current decompression algorithm.
+ */
+extern char const *zpq_decompress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Parse the compression setting. Returns:
+ * - 1 if the compression setting is valid
+ * - 0 if the compression setting is valid but disabled
+ * - -1 if the compression setting is invalid
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or if compression is disabled, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+extern int
+ zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Serialize the compressors array to string so it can be transmitted to the other side during the compression startup.
+ * For example, for array of two compressors (zstd, level 1), (zlib, level 2) resulting string would look like "zstd:1,zlib:2".
+ * Returns the resulting string.
+ */
+extern char
+ *zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors);
+
+/* Deserialize the compressors string received during the compression setup to a compressors array.
+ * For example, for string "zstd:1,zlib:2" compressors would be populated with 2 elements: (zstd, level 1), (zlib, level 2).
+ * Returns:
+ * - true if the compressors string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or string is empty, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+bool
+ zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Return the currently enabled compression algorithms */
+char *zpq_algorithms(ZpqStream * zpq);
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index dd3e5efba3..774d4638bd 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -188,6 +188,9 @@ typedef struct Port
int keepalives_count;
int tcp_user_timeout;
+ char *compression_algorithms; /* Compression algorithms supported by
+ * client */
+
/*
* GSSAPI structures.
*/
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index f0786e08b4..d568a056bc 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -71,6 +71,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port *port);
extern int pq_getbytes(char *s, size_t len);
extern void pq_startmsgread(void);
extern void pq_endmsgread(void);
diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h
index ed26ad2256..21787e3006 100644
--- a/src/include/libpq/pqcomm.h
+++ b/src/include/libpq/pqcomm.h
@@ -137,6 +137,9 @@ typedef uint32 PacketLen;
extern bool Db_user_namespace;
+/* List of allowed compression algorithms */
+extern char *libpq_compress_algorithms;
+
/*
* In protocol 3.0 and later, the startup packet length is not fixed, but
* we set an arbitrary limit on it anyway. This is just to prevent simple
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 8217d0cb6b..0e4e457dd6 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -144,6 +144,12 @@ typedef struct PgBackendStatus
/* application name; MUST be null-terminated */
char *st_appname;
+ /* client-server traffic information */
+ uint64 st_rx_raw_bytes;
+ uint64 st_tx_raw_bytes;
+ uint64 st_rx_compressed_bytes;
+ uint64 st_tx_compressed_bytes;
+
/*
* Current command string; MUST be null-terminated. Note that this string
* possibly is truncated in the middle of a multi-byte character. As
@@ -301,6 +307,7 @@ extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
+extern void pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes);
extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
int buflen);
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 844c95d47d..1c11acc2c5 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -29,6 +29,20 @@ endif
# The MSVC build system scrapes OBJS from this file. If you change any of
# the conditional additions of files to OBJS, update Mkvcbuild.pm to match.
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+SHLIB_LINK += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
+# We can't use Makefile variables here because the MSVC build system scrapes
+# OBJS from this file.
+
+
OBJS = \
$(WIN32RES) \
fe-auth-scram.o \
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index e8bcc88370..399169c26d 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -186,3 +186,5 @@ PQpipelineStatus 183
PQsetTraceFlags 184
PQmblenBounded 185
PQsendFlushRequest 186
+PQcompression 187
+PQreadPending 188
\ No newline at end of file
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index fe3af855d8..42131abbb9 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -24,6 +24,7 @@
#include "common/ip.h"
#include "common/link-canary.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "common/string.h"
#include "fe-auth.h"
#include "libpq-fe.h"
@@ -339,6 +340,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "PGCOMPRESSION", "off", NULL,
+ "Libpq-compression", "", 16,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 15, /* sizeof("prefer-standby") = 15 */
@@ -447,6 +452,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zpqStream);
+ conn->zpqStream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -1394,6 +1403,36 @@ connectOptions2(PGconn *conn)
goto oom_error;
}
+ /*
+ * validate compression option
+ */
+ if (conn->compression && conn->compression[0])
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ int rc = zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors);
+
+ if (rc == -1)
+ {
+ conn->status = CONNECTION_BAD;
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("invalid %s value: \"%s\"\n"),
+ "compression", conn->compression);
+ return false;
+ }
+
+ if (rc == 1 && n_compressors == 0)
+ {
+ conn->status = CONNECTION_BAD;
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("no supported algorithms found, %s value: \"%s\"\n"),
+ "compression", conn->compression);
+ return false;
+ }
+
+ free(compressors);
+ }
+
/*
* validate target_session_attrs option, and set target_server_type
*/
@@ -3260,11 +3299,14 @@ keep_going: /* We will come back to here until there is
}
/*
- * Validate message type: we expect only an authentication
- * request or an error here. Anything else probably means
- * it's not Postgres on the other end at all.
+ * Validate message type. We expect only:
+ * - authentication request ('R')
+ * - error ('E')
+ * - protocol compression acknowledgment ('z')
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
{
appendPQExpBuffer(&conn->errorMessage,
libpq_gettext("expected authentication request from server, but received %c\n"),
@@ -3421,6 +3463,70 @@ keep_going: /* We will come back to here until there is
goto error_return;
}
+ if (beresp == 'z') /* Switch on compression */
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ char *resp = malloc(msgLength);
+
+ pqGetnchar(resp, msgLength, conn);
+
+ if (!zpq_deserialize_compressors(resp, &compressors, &n_compressors))
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server returned unrecognized compression setting: %s\n"),
+ resp);
+ free(resp);
+ goto error_return;
+ }
+ free(resp);
+
+ if (n_compressors == 0)
+ {
+ /*
+ * If there are no compressors returned, it means that
+ * the server rejected all the proposed compression
+ * algorithms. Report an error and exit.
+ */
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server rejected protocol compression\n"));
+ goto error_return;
+ }
+
+ Assert(!conn->zpqStream);
+ conn->zpqStream = zpq_create(compressors, n_compressors,
+ (zpq_tx_func) pqsecure_write, (zpq_rx_func) pqsecure_read,
+ conn,
+ &conn->inBuffer[conn->inCursor],
+ conn->inEnd - conn->inCursor);
+ if (!conn->zpqStream)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("failed to initialize compression\n"));
+ free(compressors);
+ goto error_return;
+ }
+ /* reset buffer */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+ else if (conn->n_compressors > 0 && !conn->zpqStream)
+ {
+ /*
+ * Despite the client requesting the compression, the
+ * backend did not reply with the CompressionACK message.
+ * This case covers either no reply at all or
+ * NegotiateProtocolVersion reply. If the backend supports
+ * the protocol compression feature, it should reply with
+ * CompressionACK in any case, even in case of compression
+ * request rejection. If the backend did not reply with
+ * the CompressionACK message, it means that it does not
+ * support this feature. Report an error and exit.
+ */
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server does not support protocol compression\n"));
+ goto error_return;
+ }
+
/* It is an authentication request. */
conn->auth_req_received = true;
@@ -4085,6 +4191,8 @@ freePGconn(PGconn *conn)
free(conn->dbName);
if (conn->replication)
free(conn->replication);
+ if (conn->compression)
+ free(conn->compression);
if (conn->pguser)
free(conn->pguser);
if (conn->pgpass)
@@ -6632,6 +6740,15 @@ PQuser(const PGconn *conn)
return conn->pguser;
}
+char *
+PQcompression(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+
+ return zpq_algorithms(conn->zpqStream);
+}
+
char *
PQpass(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 59121873d2..c5ec552354 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2010,9 +2010,7 @@ PQgetResult(PGconn *conn)
* EOF indication. We expect therefore that this won't result in any
* undue delay in reporting a previous write failure.)
*/
- if (flushResult ||
- pqWait(true, false, conn) ||
- pqReadData(conn) < 0)
+ if (flushResult || pqWait(true, false, conn) || pqReadData(conn) < 0)
{
/*
* conn->errorMessage has been set by pqWait or pqReadData. We
@@ -3785,6 +3783,12 @@ pqPipelineFlush(PGconn *conn)
return 0;
}
+int
+PQreadPending(PGconn *conn)
+{
+ return pqReadPending(conn);
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 7fcfe08fd2..1662a30b3e 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -53,12 +53,24 @@
#include "pg_config_paths.h"
#include "port/pg_bswap.h"
+#include <common/zpq_stream.h>
+
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
+/*
+ * Use zpq_read if compression is switched on
+ */
+#define pq_read_conn(conn) \
+ (conn->zpqStream \
+ ? zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd, false) \
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd))
+
/*
* PQlibVersion: return the libpq version number
*/
@@ -617,10 +629,17 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -712,10 +731,18 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
+
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -826,12 +853,18 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered_tx(conn->zpqStream))
{
int sent;
+ size_t processed = 0;
+ /*
+ * Use zpq_write if compression is switched on
+ */
+ sent = conn->zpqStream
+ ? zpq_write(conn->zpqStream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -839,8 +872,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+: pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -896,7 +932,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered_tx(conn->zpqStream))
{
/*
* We didn't send it all, wait till we can send more.
@@ -964,7 +1000,7 @@ pqSendSome(PGconn *conn, int len)
int
pqFlush(PGconn *conn)
{
- if (conn->outCount > 0)
+ if (conn->outCount > 0 || zpq_buffered_tx(conn->zpqStream))
{
if (conn->Pfdebug)
fflush(conn->Pfdebug);
@@ -989,6 +1025,8 @@ pqFlush(PGconn *conn)
int
pqWait(int forRead, int forWrite, PGconn *conn)
{
+ if (forRead && conn->inCursor < conn->inEnd)
+ return 0;
return pqWaitTimed(forRead, forWrite, conn, (time_t) -1);
}
@@ -1046,6 +1084,9 @@ pqWriteReady(PGconn *conn)
*
* If SSL is in use, the SSL buffer is checked prior to checking the socket
* for read data directly.
+ *
+ * If ZPQ stream is in use, the ZPQ buffer is checked prior to checking
+ * the socket for read data directly.
*/
static int
pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
@@ -1061,14 +1102,10 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return -1;
}
-#ifdef USE_SSL
- /* Check for SSL library buffering read bytes */
- if (forRead && conn->ssl_in_use && pgtls_read_pending(conn))
+ if (forRead && (pqReadPending(conn) > 0))
{
- /* short-circuit the select */
return 1;
}
-#endif
/* We will retry as long as we get EINTR */
do
@@ -1088,6 +1125,33 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return result;
}
+/*
+ * Check if there is some data pending in ZPQ / SSL read buffers.
+ * Returns -1 on failure, 0 if no, 1 if yes.
+ */
+int
+pqReadPending(PGconn *conn)
+{
+ if (!conn)
+ return -1;
+
+ /* check for ZPQ stream buffered read bytes */
+ if (zpq_buffered_rx(conn->zpqStream))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+
+#ifdef USE_SSL
+ /* Check for SSL library buffering read bytes */
+ if (conn->ssl_in_use && pgtls_read_pending(conn))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+#endif
+ return 0;
+}
/*
* Check a file descriptor for read and/or write data, possibly waiting.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 26dbeaed97..b096783c57 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -71,6 +71,19 @@ pqParseInput3(PGconn *conn)
*/
for (;;)
{
+ /*
+ * Read the buffered compressed data w/o blocking
+ */
+ if (conn->zpqStream && pqReadPending(conn) && (conn->inBufSize - conn->inEnd > 0))
+ {
+ int rc = zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, conn->inBufSize - conn->inEnd, true);
+
+ if (rc > 0)
+ {
+ conn->inEnd += rc;
+ }
+ }
+
/*
* Try to read a message. First get the type code and length. Return
* if not enough data.
@@ -1708,7 +1721,7 @@ pqGetCopyData3(PGconn *conn, char **buffer, int async)
if (msgLength == 0)
{
/* Don't block if async read requested */
- if (async)
+ if (async && !pqReadPending(conn))
return 0;
/* Need to load more data */
if (pqWait(true, false, conn) ||
@@ -2174,6 +2187,50 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
return startpacket;
}
+/*
+ * Build comma-separated list of compression algorithms requested by client.
+ * It can be either explicitly specified by user in connection string, or
+ * include all algorithms supported by client library.
+ * This function returns true if the compression string is successfully parsed and
+ * stores a comma-separated list of algorithms in *client_compressors.
+ * If compression is disabled, then NULL is assigned to *client_compressors.
+ * Also it creates an array of compressor descriptors, each element of which corresponds to
+ * the corresponding algorithm name in *client_compressors list. This array is stored in PGconn
+ * and is used during handshake when a compression acknowledgment response is received from the server.
+ */
+static bool
+build_compressors_list(PGconn *conn, char **client_compressors, bool build_descriptors)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors))
+ {
+ return false;
+ }
+
+ *client_compressors = NULL;
+ if (build_descriptors)
+ {
+ conn->compressors = compressors;
+ conn->n_compressors = n_compressors;
+ }
+
+ if (n_compressors == 0)
+ {
+ /* no compressors available, return */
+ return true;
+ }
+
+ *client_compressors = zpq_serialize_compressors(compressors, n_compressors);
+
+ if (!build_descriptors)
+ {
+ free(compressors);
+ }
+ return true;
+}
+
/*
* Build a startup packet given a filled-in PGconn structure.
*
@@ -2220,6 +2277,18 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("replication", conn->replication);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
+ if (conn->compression && conn->compression[0])
+ {
+ char *client_compression_algorithms;
+
+ if (build_compressors_list((PGconn *) conn, &client_compression_algorithms, packet == NULL))
+ {
+ if (client_compression_algorithms)
+ {
+ ADD_STARTUP_OPTION("_pq_.compression", client_compression_algorithms);
+ }
+ }
+ }
if (conn->send_appname)
{
/* Use appname if present, otherwise use fallback */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 20eb855abc..be68fff574 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -340,6 +340,7 @@ extern char *PQhostaddr(const PGconn *conn);
extern char *PQport(const PGconn *conn);
extern char *PQtty(const PGconn *conn);
extern char *PQoptions(const PGconn *conn);
+extern char *PQcompression(const PGconn *conn);
extern ConnStatusType PQstatus(const PGconn *conn);
extern PGTransactionStatusType PQtransactionStatus(const PGconn *conn);
extern const char *PQparameterStatus(const PGconn *conn,
@@ -498,6 +499,9 @@ extern PGPing PQpingParams(const char *const *keywords,
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
+extern int
+ PQreadPending(PGconn *conn);
+
/*
* "Fast path" interface --- not really recommended for application
* use
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index fcce13843e..47b9cdadbf 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -40,6 +40,7 @@
/* include stuff common to fe and be */
#include "getaddrinfo.h"
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "fe-auth-sasl.h"
#include "pqexpbuffer.h"
@@ -340,6 +341,7 @@ typedef struct pg_conn_host
* found in password file. */
} pg_conn_host;
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -392,6 +394,14 @@ struct pg_conn
* "sspi") */
char *ssl_min_protocol_version; /* minimum TLS protocol version */
char *ssl_max_protocol_version; /* maximum TLS protocol version */
+
+ char *compression; /* stream compression (boolean value, "any" or
+ * list of compression algorithms separated by
+ * comma) */
+ zpq_compressor *compressors; /* descriptors of compression algorithms
+ * chosen by client */
+ unsigned n_compressors; /* size of compressors array */
+
char *target_session_attrs; /* desired session properties */
/* Optional file to write trace info to */
@@ -572,6 +582,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream *zpqStream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
@@ -697,6 +710,7 @@ extern int pqWaitTimed(int forRead, int forWrite, PGconn *conn,
time_t finish_time);
extern int pqReadReady(PGconn *conn);
extern int pqWriteReady(PGconn *conn);
+extern int pqReadPending(PGconn *conn);
/* === in fe-secure.c === */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d652f7b5fb..f171f92b44 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1765,7 +1765,7 @@ pg_stat_activity| SELECT s.datid,
s.query_id,
s.query,
s.backend_type
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_all_indexes| SELECT c.oid AS relid,
@@ -1877,8 +1877,14 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_auth AS gss_authenticated,
s.gss_princ AS principal,
s.gss_enc AS encrypted
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
+pg_stat_network_traffic| SELECT s.pid,
+ s.rx_raw_bytes,
+ s.tx_raw_bytes,
+ s.rx_compressed_bytes,
+ s.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
@@ -2047,7 +2053,7 @@ pg_stat_replication| SELECT s.pid,
w.sync_priority,
w.sync_state,
w.reply_time
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_replication_slots| SELECT s.slot_name,
@@ -2081,7 +2087,7 @@ pg_stat_ssl| SELECT s.pid,
s.ssl_client_dn AS client_dn,
s.ssl_client_serial AS client_serial,
s.ssl_issuer_dn AS issuer_dn
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ec3546d0c0..92dfb27a41 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -130,7 +130,7 @@ sub mkvcbuild
keywords.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c
rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c
- username.c wait_error.c wchar.c);
+ username.c wait_error.c wchar.c z_stream.c zpq_stream.c);
if ($solution->{options}->{openssl})
{
--
2.30.2
=
0002-Add-ZSTD-support.patch (application/octet-stream, 17.8K) download
From 1a18a30b47fb1ff5199bb3fbff952f64bdd93f19 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:59:01 +0500
Subject: [PATCH 2/3] Add ZSTD support
---
configure | 214 +++++++++++++++++++++++++++++++
configure.ac | 36 ++++++
src/Makefile.global.in | 1 +
src/backend/Makefile | 4 +
src/common/z_stream.c | 228 ++++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
src/interfaces/libpq/Makefile | 5 +
src/tools/msvc/Solution.pm | 1 +
8 files changed, 492 insertions(+)
diff --git a/configure b/configure
index 3f2aea0d7d..fbf93c1fbb 100755
--- a/configure
+++ b/configure
@@ -699,6 +699,9 @@ with_gnu_ld
LD
LDFLAGS_SL
LDFLAGS_EX
+ZSTD_LIBS
+ZSTD_CFLAGS
+with_zstd
LZ4_LIBS
LZ4_CFLAGS
with_lz4
@@ -868,6 +871,7 @@ with_libxslt
with_system_tzdata
with_zlib
with_lz4
+with_zstd
with_gnu_ld
with_ssl
with_openssl
@@ -897,6 +901,8 @@ XML2_CFLAGS
XML2_LIBS
LZ4_CFLAGS
LZ4_LIBS
+ZSTD_CFLAGS
+ZSTD_LIBS
LDFLAGS_EX
LDFLAGS_SL
PERL
@@ -1576,6 +1582,7 @@ Optional Packages:
use system time zone data in DIR
--without-zlib do not use Zlib
--with-lz4 build with LZ4 support
+ --with-zstd build with ZSTD support
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-ssl=LIB use LIB for SSL/TLS support (openssl)
--with-openssl obsolete spelling of --with-ssl=openssl
@@ -1605,6 +1612,8 @@ Some influential environment variables:
XML2_LIBS linker flags for XML2, overriding pkg-config
LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config
LZ4_LIBS linker flags for LZ4, overriding pkg-config
+ ZSTD_CFLAGS C compiler flags for ZSTD, overriding pkg-config
+ ZSTD_LIBS linker flags for ZSTD, overriding pkg-config
LDFLAGS_EX extra linker flags for linking executables only
LDFLAGS_SL extra linker flags for linking shared libraries only
PERL Perl program
@@ -9033,6 +9042,145 @@ fi
done
fi
+#
+# Zstd
+#
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with ZSTD support" >&5
+$as_echo_n "checking whether to build with ZSTD support... " >&6; }
+
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+ withval=$with_zstd;
+ case $withval in
+ yes)
+
+$as_echo "#define HAVE_LIBZSTD 1" >>confdefs.h
+
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_zstd=no
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_zstd" >&5
+$as_echo "$with_zstd" >&6; }
+
+
+if test "$with_zstd" = yes; then
+
+pkg_failed=no
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libzstd >= 1.0.0" >&5
+$as_echo_n "checking for libzstd >= 1.0.0... " >&6; }
+
+if test -n "$ZSTD_CFLAGS"; then
+ pkg_cv_ZSTD_CFLAGS="$ZSTD_CFLAGS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_CFLAGS=`$PKG_CONFIG --cflags "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+if test -n "$ZSTD_LIBS"; then
+ pkg_cv_ZSTD_LIBS="$ZSTD_LIBS"
+ elif test -n "$PKG_CONFIG"; then
+ if test -n "$PKG_CONFIG" && \
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"libzstd >= 1.0.0\""; } >&5
+ ($PKG_CONFIG --exists --print-errors "libzstd >= 1.0.0") 2>&5
+ ac_status=$?
+ $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5
+ test $ac_status = 0; }; then
+ pkg_cv_ZSTD_LIBS=`$PKG_CONFIG --libs "libzstd >= 1.0.0" 2>/dev/null`
+ test "x$?" != "x0" && pkg_failed=yes
+else
+ pkg_failed=yes
+fi
+ else
+ pkg_failed=untried
+fi
+
+
+
+if test $pkg_failed = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+
+if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then
+ _pkg_short_errors_supported=yes
+else
+ _pkg_short_errors_supported=no
+fi
+ if test $_pkg_short_errors_supported = yes; then
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ else
+ ZSTD_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "libzstd >= 1.0.0" 2>&1`
+ fi
+ # Put the nasty error message in config.log where it belongs
+ echo "$ZSTD_PKG_ERRORS" >&5
+
+ as_fn_error $? "Package requirements (libzstd >= 1.0.0) were not met:
+
+$ZSTD_PKG_ERRORS
+
+Consider adjusting the PKG_CONFIG_PATH environment variable if you
+installed software in a non-standard prefix.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details." "$LINENO" 5
+elif test $pkg_failed = untried; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it
+is in your PATH or set the PKG_CONFIG environment variable to the full
+path to pkg-config.
+
+Alternatively, you may set the environment variables ZSTD_CFLAGS
+and ZSTD_LIBS to avoid the need to call pkg-config.
+See the pkg-config man page for more details.
+
+To get pkg-config, see <http://pkg-config.freedesktop.org/>.
+See \`config.log' for more details" "$LINENO" 5; }
+else
+ ZSTD_CFLAGS=$pkg_cv_ZSTD_CFLAGS
+ ZSTD_LIBS=$pkg_cv_ZSTD_LIBS
+ { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -12523,6 +12671,58 @@ fi
fi
+if test "$with_zstd" = yes; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_decompressStream in -lzstd" >&5
+$as_echo_n "checking for ZSTD_decompressStream in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_decompressStream+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_decompressStream ();
+int
+main ()
+{
+return ZSTD_decompressStream ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_zstd_ZSTD_decompressStream=yes
+else
+ ac_cv_lib_zstd_ZSTD_decompressStream=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_decompressStream" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_decompressStream" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_decompressStream" = xyes; then :
+ cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+ LIBS="-lzstd $LIBS"
+
+else
+ as_fn_error $? "zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory." "$LINENO" 5
+fi
+
+fi
+
if test "$enable_spinlocks" = yes; then
$as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -13854,6 +14054,20 @@ fi
done
+fi
+
+if test "$with_zstd" = yes; then
+ ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+ as_fn_error $? "zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support." "$LINENO" 5
+fi
+
+
fi
if test "$with_gssapi" = yes ; then
diff --git a/configure.ac b/configure.ac
index 95287705f6..7759ff3613 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1056,6 +1056,28 @@ if test "$with_lz4" = yes; then
done
fi
+#
+# Zstd
+#
+AC_MSG_CHECKING([whether to build with ZSTD support])
+PGAC_ARG_BOOL(with, zstd, no, [build with ZSTD support], [AC_DEFINE([HAVE_LIBZSTD], 1, [Define to 1 to build with ZSTD support. (--with-zstd)])])
+AC_MSG_RESULT([$with_zstd])
+AC_SUBST(with_zstd)
+
+if test "$with_zstd" = yes; then
+ PKG_CHECK_MODULES(ZSTD, [libzstd >= 1.0.0])
+ for pgac_option in $ZSTD_CFLAGS; do
+ case $pgac_option in
+ -I*|-D*) CPPFLAGS="$CPPFLAGS $pgac_option";;
+ esac
+ done
+ for pgac_option in $ZSTD_LIBS; do
+ case $pgac_option in
+ -L*) LDFLAGS="$LDFLAGS $pgac_option";;
+ esac
+ done
+fi
+
#
# Assignments
#
@@ -1242,6 +1264,13 @@ failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_LIB(zstd, ZSTD_decompressStream, [],
+ [AC_MSG_ERROR([zstd library not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.])])
+fi
+
if test "$enable_spinlocks" = yes; then
AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
else
@@ -1488,6 +1517,13 @@ if test "$with_lz4" = yes; then
AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])
fi
+if test "$with_zstd" = yes; then
+ AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([zstd header not found
+If you have zstd already installed, see config.log for details on the
+failure. It is possible the compiler isn't looking in the proper directory.
+Use --without-zstd to disable zstd support.])])
+fi
+
if test "$with_gssapi" = yes ; then
AC_CHECK_HEADERS(gssapi/gssapi.h, [],
[AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])])
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 79ed589ce9..71e14ee68f 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -197,6 +197,7 @@ with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
with_lz4 = @with_lz4@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index dab379b412..94cae64831 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -54,6 +54,10 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
ifeq ($(with_lz4),yes)
LIBS += -llz4
endif
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index f8adbfced2..771764d340 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -89,6 +89,231 @@ struct ZStream
bool not_flushed;
};
+#if HAVE_LIBZSTD
+
+#include <stdlib.h>
+#include <zstd.h>
+
+/*
+ * Maximum allowed back-reference distance, expressed as power of 2.
+ * This setting controls max compressor/decompressor window size.
+ * More details https://github.com/facebook/zstd/blob/v1.4.7/lib/zstd.h#L536
+ */
+#define ZSTD_WINDOWLOG_LIMIT 23 /* set max window size to 8MB */
+
+
+typedef struct ZS_ZSTD_CStream
+{
+ ZSTD_CStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_CStream;
+
+typedef struct ZS_ZSTD_DStream
+{
+ ZSTD_DStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_DStream;
+
+static void *
+zstd_create_compressor(int level)
+{
+ size_t rc;
+ ZS_ZSTD_CStream *c_stream = (ZS_ZSTD_CStream *) malloc(sizeof(ZS_ZSTD_CStream));
+
+ c_stream->stream = ZSTD_createCStream();
+ rc = ZSTD_initCStream(c_stream->stream, level);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeCStream(c_stream->stream);
+ free(c_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_CCtx_setParameter(c_stream->stream, ZSTD_c_windowLog, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ c_stream->error = NULL;
+ return c_stream;
+}
+
+static void *
+zstd_create_decompressor()
+{
+ size_t rc;
+ ZS_ZSTD_DStream *d_stream = (ZS_ZSTD_DStream *) malloc(sizeof(ZS_ZSTD_DStream));
+
+ d_stream->stream = ZSTD_createDStream();
+ rc = ZSTD_initDStream(d_stream->stream);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeDStream(d_stream->stream);
+ free(d_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_DCtx_setParameter(d_stream->stream, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ d_stream->error = NULL;
+ return d_stream;
+}
+
+static ssize_t
+zstd_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+ size_t rc;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ rc = ZSTD_decompressStream(ds->stream, &out, &in);
+
+ *src_processed = in.pos;
+ *dst_processed = out.pos;
+ if (ZSTD_isError(rc))
+ {
+ ds->error = ZSTD_getErrorName(rc);
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ if (rc == 0)
+ {
+ return ZS_STREAM_END;
+ }
+
+ if (out.pos == out.size)
+ {
+ /*
+ * if `output.pos == output.size`, there might be some data left
+ * within internal buffers
+ */
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ if (in.pos < src_size) /* Has something to compress in input buffer */
+ {
+ size_t rc = ZSTD_compressStream(cs->stream, &out, &in);
+
+ *dst_processed = out.pos;
+ *src_processed = in.pos;
+ if (ZSTD_isError(rc))
+ {
+ cs->error = ZSTD_getErrorName(rc);
+ return ZS_COMPRESS_ERROR;
+ }
+ }
+
+ if (in.pos == src_size) /* All data is compressed: flush internal zstd
+ * buffer */
+ {
+ size_t tx_not_flushed = ZSTD_flushStream(cs->stream, &out);
+
+ *dst_processed = out.pos;
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ size_t tx_not_flushed;
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_outBuffer output;
+
+ output.dst = dst;
+ output.pos = 0;
+ output.size = dst_size;
+
+ do
+ {
+ tx_not_flushed = ZSTD_endStream(cs->stream, &output);
+ } while ((tx_not_flushed > 0) && (output.pos < output.size));
+
+ *dst_processed = output.pos;
+
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static void
+zstd_free_compressor(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ ZSTD_freeCStream(cs->stream);
+ free(cs);
+ }
+}
+
+static void
+zstd_free_decompressor(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ ZSTD_freeDStream(ds->stream);
+ free(ds);
+ }
+}
+
+static char const *
+zstd_compress_error(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ return cs->error;
+}
+
+static char const *
+zstd_decompress_error(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ return ds->error;
+}
+
+static char const *
+zstd_name(void)
+{
+ return "zstd";
+}
+
+#endif
+
#if HAVE_LIBZ
#include <stdlib.h>
@@ -430,6 +655,9 @@ no_compression_name(void)
*/
static ZAlgorithm const zs_algorithms[] =
{
+#if HAVE_LIBZSTD
+ {zstd_name, zstd_create_compressor, zstd_create_decompressor, zstd_decompress, zstd_compress, zstd_free_compressor, zstd_free_decompressor, zstd_compress_error, zstd_decompress_error, zstd_end},
+#endif
#if HAVE_LIBZ
{zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
#endif
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 9d9bd6b9ef..52c66d2e61 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -352,6 +352,9 @@
/* Define to 1 if you have the `z' library (-lz). */
#undef HAVE_LIBZ
+/* Define to 1 if you have the `zstd' library (-lzstd). */
+#undef HAVE_LIBZSTD
+
/* Define to 1 if you have the `link' function. */
#undef HAVE_LINK
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 1c11acc2c5..00be5ec1cc 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -34,6 +34,11 @@ LIBS += -llz4
SHLIB_LINK += -llz4
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
ifeq ($(with_zlib),yes)
LIBS += -lz
SHLIB_LINK += -lz
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index e47c2d648c..e402e3af15 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -311,6 +311,7 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
--
2.30.2
=
0003-Turn-on-zlib-compression-for-CI-test-runs.patch (application/octet-stream, 9.9K) download
From 115173017d45eec3e8d07e6c21bc6aee8e7f8c4e Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Mon, 17 Jan 2022 17:46:03 +0500
Subject: [PATCH 3/3] Turn on zlib compression for CI test runs
---
.cirrus.yml | 2 +
src/backend/utils/misc/guc.c | 2 +-
src/common/zpq_stream.c | 27 +---
src/interfaces/libpq/fe-connect.c | 2 +-
src/tools/ci/install_zlib.bat | 201 +++++++++++++++++++++++++++
src/tools/ci/windows_build_config.pl | 1 +
src/tools/msvc/Solution.pm | 2 +-
7 files changed, 211 insertions(+), 26 deletions(-)
create mode 100644 src/tools/ci/install_zlib.bat
diff --git a/.cirrus.yml b/.cirrus.yml
index 677bdf0e65..297ac51651 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -381,6 +381,8 @@ task:
configure_script:
# copy errors out when using forward slashes
+ - choco install -y cmake
+ - ./src/tools/ci/install_zlib.bat
- copy src\tools\ci\windows_build_config.pl src\tools\msvc\config.pl
- vcvarsall x64
- perl src/tools/msvc/mkvcbuild.pl
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 9d8fd9b817..11a43ba668 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -4601,7 +4601,7 @@ static struct config_string ConfigureNamesString[] =
NULL
},
&libpq_compress_algorithms,
- "off",
+ "zlib",
check_libpq_compression, NULL, NULL
},
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
index a712200e45..de00a82896 100644
--- a/src/common/zpq_stream.c
+++ b/src/common/zpq_stream.c
@@ -156,24 +156,8 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
for (i = 0; i < 256; i++)
{
- zpq->compressor_by_msg_type[i] = -1;
- }
-
- for (i = 0; i < zpq->n_compressors; i++)
- {
- /* compress CopyData, DataRow and Query messages */
- if (zpq->compressor_by_msg_type['d'] == -1)
- {
- zpq->compressor_by_msg_type['d'] = i;
- }
- if (zpq->compressor_by_msg_type['D'] == -1)
- {
- zpq->compressor_by_msg_type['D'] = i;
- }
- if (zpq->compressor_by_msg_type['Q'] == -1)
- {
- zpq->compressor_by_msg_type['Q'] = i;
- }
+ /* Compress all messages during the CI test runs */
+ zpq->compressor_by_msg_type[i] = 0;
}
}
@@ -186,11 +170,8 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
static inline int
zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
{
- if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
- {
- return zpq->compressor_by_msg_type[(unsigned char) msg_type];
- }
- return -1;
+ /* Compress messages with any length during the CI test runs */
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
}
/*
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 42131abbb9..2cb0b2f4f0 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -340,7 +340,7 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
- {"compression", "PGCOMPRESSION", "off", NULL,
+ {"compression", "PGCOMPRESSION", "zlib", NULL,
"Libpq-compression", "", 16,
offsetof(struct pg_conn, compression)},
diff --git a/src/tools/ci/install_zlib.bat b/src/tools/ci/install_zlib.bat
new file mode 100644
index 0000000000..e8419377c8
--- /dev/null
+++ b/src/tools/ci/install_zlib.bat
@@ -0,0 +1,201 @@
+@echo off
+setlocal EnableDelayedExpansion
+
+set ORIGIN=%cd%
+
+:: Configuration
+set VERSION=1.2.11
+set VER=%VERSION:.=%
+set URL=https://zlib.net/zlib%VER%.zip
+set CMAKE_VS_PLATFORM_NAME=x64
+set TMPDIR=%HOMEDRIVE%%HOMEPATH%\zlib_tmp
+set SRC_DIR=%TMPDIR%\zlib-%VERSION%
+set ZIPFILE=zlib%VER%.zip
+set ABS_ZIPFILE=%TMPDIR%\%ZIPFILE%
+set BUILD_DIR=%SRC_DIR%\build
+set LOGFILE=%TMPDIR%\zlib_install.log
+
+echo [0/6] Library(zlib==%VERSION%)
+
+if not exist %TMPDIR% (mkdir %TMPDIR% && cd /d %TMPDIR% || exit /B 1)
+call :cleanup_src
+call :cleanup_log
+copy /y nul %LOGFILE% >nul 2>&1
+call :log_sysinfo >>%LOGFILE% 2>&1
+
+if not defined ARCH (
+ set ARCH=x64
+)
+set GENERATOR=-DCMAKE_GENERATOR_PLATFORM=%ARCH%
+if "%ARCH%" == "x64" (
+ set PREFIX=-DCMAKE_INSTALL_PREFIX="C:\zlib"
+) else (
+ set PREFIX=-DCMAKE_INSTALL_PREFIX="C:\zlib"
+)
+
+echo|set /p="[1/6] Checking cmake... "
+call :setup_cmake_path >>%LOGFILE% 2>&1
+if not defined CMAKE (
+ call :failed
+ echo[
+ echo Please, install CMAKE: https://cmake.org/download/
+ exit /B 1
+) else (echo done.)
+
+echo|set /p="[2/6] Downloading... "
+echo Fetching %URL% >>%LOGFILE% 2>&1
+cd /d %TMPDIR% && call :winget "%URL%" >>%LOGFILE% 2>&1
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1) else (echo done.)
+
+echo|set /p="[3/6] Extracting... "
+powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('%ABS_ZIPFILE%', '%TMPDIR%'); }" >>%LOGFILE% 2>&1
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1) else (echo done.)
+
+echo|set /p="[4/6] Fixing CMakeLists.txt... "
+set OLDSTR=RUNTIME DESTINATION ""\${INSTALL_BIN_DIR}\""
+set NEWSTR=RUNTIME DESTINATION ""bin\""
+call :search_replace "%OLDSTR%" "%NEWSTR%"
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1)
+
+set OLDSTR=ARCHIVE DESTINATION ""\${INSTALL_LIB_DIR}\""
+set NEWSTR=ARCHIVE DESTINATION ""lib\""
+call :search_replace "%OLDSTR%" "%NEWSTR%"
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1)
+
+set OLDSTR=LIBRARY DESTINATION ""\${INSTALL_LIB_DIR}\""
+set NEWSTR=LIBRARY DESTINATION ""lib\""
+call :search_replace "%OLDSTR%" "%NEWSTR%"
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1)
+
+set OLDSTR=DESTINATION ""\${INSTALL_INC_DIR}\""
+set NEWSTR=DESTINATION ""include\""
+call :search_replace "%OLDSTR%" "%NEWSTR%"
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1) else (echo done.)
+
+rd /S /Q %BUILD_DIR% >nul 2>&1
+mkdir %BUILD_DIR% && cd /d %BUILD_DIR%
+
+echo|set /p="[5/6] Configuring... "
+"%CMAKE%" .. -DCMAKE_BUILD_TYPE=Release %PREFIX% %GENERATOR% >>%LOGFILE% 2>&1
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1) else (echo done.)
+
+echo [6/6] Compiling and installing...
+"%CMAKE%" --build . --config Release --target install
+if %ERRORLEVEL% NEQ 0 (call :failed && exit /B 1) else (echo done.)
+
+call :cleanup_src
+call :goto_origin
+
+echo Details can be found at %LOGFILE%.
+
+@echo on
+@goto :eof
+
+:setup_cmake_path
+where cmake.exe
+cmake.exe /? 2> NUL 1> NUL
+if not %ERRORLEVEL%==9009 (echo cmake.exe is accessible by default. && set CMAKE=cmake.exe)
+if not defined CMAKE (echo cmake.exe is not accessible by default. Looking into common installation dirs.)
+if not defined CMAKE (call :setup_cmake_path_for "%PROGRAMFILES%\CMake\bin")
+if not defined CMAKE (call :setup_cmake_path_for "C:\Program Files\CMake\bin")
+if not defined CMAKE (call :setup_cmake_path_for "C:\Program Files (x86)\CMake\bin")
+@goto :eof
+
+:setup_cmake_path_for
+set DIR_PATH=%~1
+echo|set /p="Checking !DIR_PATH! ... "
+if exist !DIR_PATH!\cmake.exe (
+ echo found.
+ set CMAKE=!DIR_PATH!\cmake.exe
+) else (
+ echo not found.
+)
+@goto :eof
+
+:cleanup_src
+del /Q %ABS_ZIPFILE% >nul 2>&1
+rd /S /Q %SRC_DIR% >nul 2>&1
+@goto :eof
+
+:cleanup_log
+del /Q %LOGFILE% >nul 2>&1
+@goto :eof
+
+:log_sysinfo
+echo -- Environment variables
+set
+echo[
+
+echo -- System info
+systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
+echo[
+@goto :eof
+
+:cmakefiles_debug
+set FILE=%BUILD_DIR%\CMakeFiles\CMakeError.log
+if exist %FILE% (
+ echo -- %FILE%
+ type %FILE%
+ echo[
+)
+set FILE=%BUILD_DIR%\CMakeFiles\CMakeOutput.log
+if exist %FILE% (
+ echo -- %FILE%
+ type %FILE%
+ echo[
+)
+@goto :eof
+
+:failed
+echo FAILED.
+call :cmakefiles_debug >>%LOGFILE% 2>&1
+echo[
+echo ---------------------------------------- log begin ----------------------------------------
+type %LOGFILE%
+echo ---------------------------------------- log end ----------------------------------------
+echo LOG: %LOGFILE%
+call :cleanup_src
+call :goto_origin
+@goto :eof
+
+:goto_origin
+cd /d %ORIGIN% >nul 2>&1
+@goto :eof
+
+:search_replace
+set OLDSTR=%~1
+set NEWSTR=%~2
+cd /d %SRC_DIR%
+set CMD="(gc CMakeLists.txt) -replace '%OLDSTR%', '%NEWSTR%' | Out-File -encoding ASCII CMakeLists.txt"
+powershell -Command %CMD% >>%LOGFILE% 2>&1
+goto :eof
+
+:winget
+set URL=%~1
+for %%F in (%URL%) do set FILE=%%~nxF
+
+setlocal EnableDelayedExpansion
+set multiLine=$security_protcols = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::SystemDefault^
+
+if ([Net.SecurityProtocolType].GetMember('Tls11').Count -gt 0) {^
+
+ $security_protcols = $security_protcols -bor [Net.SecurityProtocolType]::Tls11^
+
+}^
+
+if ([Net.SecurityProtocolType].GetMember('Tls12').Count -gt 0) {^
+
+$security_protcols = $security_protcols -bor [Net.SecurityProtocolType]::Tls12^
+
+}^
+
+[Net.ServicePointManager]::SecurityProtocol = $security_protcols^
+
+Write-Host 'Downloading %URL%... ' -NoNewLine^
+
+(New-Object Net.WebClient).DownloadFile('%URL%', '%FILE%')^
+
+Write-Host 'done.'
+
+powershell -Command !multiLine!
+goto:eof
\ No newline at end of file
diff --git a/src/tools/ci/windows_build_config.pl b/src/tools/ci/windows_build_config.pl
index b0d4360c74..6629f322a5 100644
--- a/src/tools/ci/windows_build_config.pl
+++ b/src/tools/ci/windows_build_config.pl
@@ -7,6 +7,7 @@ $config->{"tap_tests"} = 1;
$config->{"asserts"} = 1;
$config->{"openssl"} = "c:/openssl/1.1/";
+$config->{"zlib"} = "c:/zlib";
$config->{"perl"} = "c:/strawberry/$ENV{DEFAULT_PERL_VERSION}/perl/";
$config->{"python"} = "c:/python/";
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index e402e3af15..387b2114d8 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -945,7 +945,7 @@ sub AddProject
if ($self->{options}->{zlib})
{
$proj->AddIncludeDir($self->{options}->{zlib} . '\include');
- $proj->AddLibrary($self->{options}->{zlib} . '\lib\zdll.lib');
+ $proj->AddLibrary($self->{options}->{zlib} . '\lib\zlibstatic.lib');
}
if ($self->{options}->{openssl})
{
--
2.30.2
=
* Re: libpq compression (part 2)
@ 2022-01-18 04:39 Justin Pryzby <pryzby@telsasoft.com>
parent: Daniil Zakhlystov <usernamedt@yandex-team.ru>
0 siblings, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-01-18 04:39 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; +Cc: pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
> > => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
> I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
> for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
> CI for some reason, but I guess it happens because of my GitHub account limitation.
I don't know about your github account, but it works for cfbot, which is now
green.
Thanks for implementing zlib for windows. Did you try this with default
compressions set to lz4 and zstd ?
The thread from 2019 is very long, and starts off with the guidance that
compression had been implemented at the wrong layer. It looks like this hasn't
changed since then. secure_read/write are passed as function pointers to the
ZPQ interface, which then calls back to them to read and flush its compression
buffers. As I understand, the suggestion was to leave the socket reads and
writes alone. And then conditionally de/compress buffers after reading /
before writing from the socket if compression was negotiated.
It's currently done like this
pq_recvbuf() => secure_read() - when compression is disabled
pq_recvbuf() => ZPQ => secure_read() - when compression is enabled
Dmitri sent a partial, POC patch which changes the de/compression to happen in
secure_read/write, which is changed to call ZPQ:
https://www.postgresql.org/message-id/CA+q6zcUPrssNaRS+FyoBsD-F2stK1Roa-4sAhFOfAjOWLziM4g@mail.gmail...
pq_recvbuf() => secure_read() => ZPQ
The same thing is true of the frontend: function pointers to
pqsecure_read/write are being passed to zpq_create, and then the ZPQ interface
called instead of the original functions. Those are the functions which read
using SSL, so they should also handle compression.
That's where SSL is handled, and it seems like the right place to handle
compression. Have you evaluated that way to do things ?
Konstantin said he put ZPQ at that layer seems to 1) avoid code duplication
between client/server; and, 2) to allow compression to happen before SSL, to
allow both (if the admin decides it's okay). But I don't see why compression
can't happen before sending to SSL, or after reading from it?
--
Justin
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-03-02 21:33 Justin Pryzby <pryzby@telsasoft.com>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-03-02 21:33 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; +Cc: pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
If there's no objection, I'd like to move this to the next CF for consideration
in PG16.
On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
> > > => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
>
> > I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
> > for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
> > CI for some reason, but I guess it happens because of my GitHub account limitation.
>
> I don't know about your github account, but it works for cfbot, which is now
> green.
>
> Thanks for implementing zlib for windows. Did you try this with default
> compressions set to lz4 and zstd ?
>
> The thread from 2019 is very long, and starts off with the guidance that
> compression had been implemented at the wrong layer. It looks like this hasn't
> changed since then. secure_read/write are passed as function pointers to the
> ZPQ interface, which then calls back to them to read and flush its compression
> buffers. As I understand, the suggestion was to leave the socket reads and
> writes alone. And then conditionally de/compress buffers after reading /
> before writing from the socket if compression was negotiated.
>
> It's currently done like this
> pq_recvbuf() => secure_read() - when compression is disabled
> pq_recvbuf() => ZPQ => secure_read() - when compression is enabled
>
> Dmitri sent a partial, POC patch which changes the de/compression to happen in
> secure_read/write, which is changed to call ZPQ:
> https://www.postgresql.org/message-id/CA+q6zcUPrssNaRS+FyoBsD-F2stK1Roa-4sAhFOfAjOWLziM4g@mail.gmail...
> pq_recvbuf() => secure_read() => ZPQ
>
> The same thing is true of the frontend: function pointers to
> pqsecure_read/write are being passed to zpq_create, and then the ZPQ interface
> called instead of the original functions. Those are the functions which read
> using SSL, so they should also handle compression.
>
> That's where SSL is handled, and it seems like the right place to handle
> compression. Have you evaluated that way to do things ?
>
> Konstantin said he put ZPQ at that layer seems to 1) avoid code duplication
> between client/server; and, 2) to allow compression to happen before SSL, to
> allow both (if the admin decides it's okay). But I don't see why compression
> can't happen before sending to SSL, or after reading from it?
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-03-03 08:50 Daniil Zakhlystov <usernamedt@yandex-team.ru>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 27+ messages in thread
From: Daniil Zakhlystov @ 2022-03-03 08:50 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
Ok, thanks
> On 3 Mar 2022, at 02:33, Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> If there's no objection, I'd like to move this to the next CF for consideration
> in PG16.
>
> On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
>> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
>>>> => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
>>
>>> I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
>>> for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
>>> CI for some reason, but I guess it happens because of my GitHub account limitation.
>>
>> I don't know about your github account, but it works for cfbot, which is now
>> green.
>>
>> Thanks for implementing zlib for windows. Did you try this with default
>> compressions set to lz4 and zstd ?
>>
>> The thread from 2019 is very long, and starts off with the guidance that
>> compression had been implemented at the wrong layer. It looks like this hasn't
>> changed since then. secure_read/write are passed as function pointers to the
>> ZPQ interface, which then calls back to them to read and flush its compression
>> buffers. As I understand, the suggestion was to leave the socket reads and
>> writes alone. And then conditionally de/compress buffers after reading /
>> before writing from the socket if compression was negotiated.
>>
>> It's currently done like this
>> pq_recvbuf() => secure_read() - when compression is disabled
>> pq_recvbuf() => ZPQ => secure_read() - when compression is enabled
>>
>> Dmitri sent a partial, POC patch which changes the de/compression to happen in
>> secure_read/write, which is changed to call ZPQ:
>> https://www.postgresql.org/message-id/CA+q6zcUPrssNaRS+FyoBsD-F2stK1Roa-4sAhFOfAjOWLziM4g@mail.gmail...
>> pq_recvbuf() => secure_read() => ZPQ
>>
>> The same thing is true of the frontend: function pointers to
>> pqsecure_read/write are being passed to zpq_create, and then the ZPQ interface
>> called instead of the original functions. Those are the functions which read
>> using SSL, so they should also handle compression.
>>
>> That's where SSL is handled, and it seems like the right place to handle
>> compression. Have you evaluated that way to do things ?
>>
>> Konstantin said he put ZPQ at that layer seems to 1) avoid code duplication
>> between client/server; and, 2) to allow compression to happen before SSL, to
>> allow both (if the admin decides it's okay). But I don't see why compression
>> can't happen before sending to SSL, or after reading from it?
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-08-02 20:53 Jacob Champion <jchampion@timescale.com>
parent: Daniil Zakhlystov <usernamedt@yandex-team.ru>
0 siblings, 1 reply; 27+ messages in thread
From: Jacob Champion @ 2022-08-02 20:53 UTC (permalink / raw)
To: Daniil Zakhlystov <usernamedt@yandex-team.ru>; Justin Pryzby <pryzby@telsasoft.com>; +Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
This entry has been waiting on author input for a while (our current
threshold is roughly two weeks), so I've marked it Returned with
Feedback.
Once you think the patchset is ready for review again, you (or any
interested party) can resurrect the patch entry by visiting
https://commitfest.postgresql.org/38/3499/
and changing the status to "Needs Review", and then changing the
status again to "Move to next CF". (Don't forget the second step;
hopefully we will have streamlined this in the near future!)
Thanks,
--Jacob
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-12 21:47 Andrey Borodin <amborodin86@gmail.com>
parent: Jacob Champion <jchampion@timescale.com>
0 siblings, 1 reply; 27+ messages in thread
From: Andrey Borodin @ 2022-11-12 21:47 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>; Justin Pryzby <pryzby@telsasoft.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
On Tue, Aug 2, 2022 at 1:53 PM Jacob Champion <jchampion@timescale.com> wrote:
>
> and changing the status to "Needs Review"
I've tried the patch, it works as advertised. The code generally is
OK, maybe some functions require comments (because at least their
neighbours have some).
Some linkers complain about zs_is_valid_impl_id() being inline while
used in other modules.
Some architectural notes:
1. Currently when the user requests compression from libpq, but the
server does not support any of the codecs client have - the connection
will be rejected by client. I think this should be configured akin to
SSL: on, try, off.
2. On the zpq_stream level of abstraction we parse a stream of bytes
to look for individual message headers. I think this is OK, just a
little bit awkward.
3. CompressionAck message can be completely replaced by ParameterStatus message.
4. Instead of sending a separate SetCompressionMethod, the
CompressedData can have its header with the index of the used method
and the necessity to restart compression context.
5. Portions of pg_stat_network_traffic can be extracted to separate
patch step to ease the review. And, actually, the scope of this view
is slightly beyond compression anyway.
What do you think?
Also, compression is a very cool and awaited feature, hope to see it
committed one day, thank you for working on this!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-13 04:04 Andrey Borodin <amborodin86@gmail.com>
parent: Andrey Borodin <amborodin86@gmail.com>
0 siblings, 1 reply; 27+ messages in thread
From: Andrey Borodin @ 2022-11-13 04:04 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>; Justin Pryzby <pryzby@telsasoft.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
On Sat, Nov 12, 2022 at 1:47 PM Andrey Borodin <amborodin86@gmail.com> wrote:
>
> I've tried the patch, it works as advertised.
While testing patch some more I observe unpleasant segfaults:
#26 0x00007fecafa1e058 in __memcpy_ssse3_back () from target:/lib64/libc.so.6
#27 0x000000000b08fda2 in lz4_decompress (d_stream=0x18cf82a0,
src=0x7feae4fa505d, src_size=92,
src_processed=0x7ffff9f4fdf8, dst=0x18b01f80, dst_size=8192,
dst_processed=0x7ffff9f4fe60)
#28 0x000000000b090624 in zs_read (zs=0x18cdfbf0, src=0x7feae4fa505d,
src_size=92, src_processed=0x7ffff9f4fdf8,
dst=0x18b01f80, dst_size=8192, dst_processed=0x7ffff9f4fe60)
#29 0x000000000b08eb8f in zpq_read_compressed_message
(zpq=0x7feae4fa5010, dst=0x18b01f80 "Q", dst_len=8192,
dst_processed=0x7ffff9f4fe60)
#30 0x000000000b08f1a9 in zpq_read (zpq=0x7feae4fa5010,
dst=0x18b01f80, dst_size=8192, noblock=false)
(gdb) select-frame 27
(gdb) info locals
ds = 0x18cf82a0
decPtr = 0x18cf8aec ""
decBytes = -87
This is the buffer overrun by decompression. I think the receive
buffer must be twice bigger than the send buffer to accommodate such
messages.
Also this portion of lz4_decompress()
Assert(decBytes > 0);
must actually be a real check and elog(ERROR,). Because clients can
intentionally compose CompressedData to blow up a server.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-15 03:44 Andrey Borodin <amborodin86@gmail.com>
parent: Andrey Borodin <amborodin86@gmail.com>
0 siblings, 1 reply; 27+ messages in thread
From: Andrey Borodin @ 2022-11-15 03:44 UTC (permalink / raw)
To: Jacob Champion <jchampion@timescale.com>; +Cc: Daniil Zakhlystov <usernamedt@yandex-team.ru>; Justin Pryzby <pryzby@telsasoft.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Andrey Borodin <x4mmm@yandex-team.ru>
On Sat, Nov 12, 2022 at 8:04 PM Andrey Borodin <amborodin86@gmail.com> wrote:
>
> While testing patch some more I observe unpleasant segfaults:
>
> #27 0x000000000b08fda2 in lz4_decompress (d_stream=0x18cf82a0,
> src=0x7feae4fa505d, src_size=92,
> (gdb) select-frame 27
> (gdb) info locals
> ds = 0x18cf82a0
> decPtr = 0x18cf8aec ""
> decBytes = -87
>
I've debugged the stuff and the problem resulted to be in wrong
message limits for Lz4 compression
+#define MESSAGE_MAX_BYTES 819200
instead of
+#define MESSAGE_MAX_BYTES 8192
Other codecs can utilize continuation of the decompression stream
using ZS_DATA_PENDING, while Lz4 cannot do this. I was going to
produce quickfix for all my lz4 findings, but it occured to me that a
patchset needs a heavy rebase. If no one shows up to fix it I'll do
that in one of the coming weekends. Either way here's a reproducer for
the coredump:
psql 'compression=lz4' -f boom.sql
Thanks!
Best regards, Andrey Borodin.
Attachments:
boom.sql (application/octet-stream, 16.0K) download
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-16 03:17 Justin Pryzby <pryzby@telsasoft.com>
parent: Andrey Borodin <amborodin86@gmail.com>
0 siblings, 1 reply; 27+ messages in thread
From: Justin Pryzby @ 2022-11-16 03:17 UTC (permalink / raw)
To: Andrey Borodin <amborodin86@gmail.com>; +Cc: Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On Mon, Nov 14, 2022 at 07:44:24PM -0800, Andrey Borodin wrote:
> patchset needs a heavy rebase. If no one shows up to fix it I'll do
Despite what its git timestamp says, this is based on the most recent
patch from January, which I've had floating around since then. It
needed to be rebased over at least:
- guc_tables patch;
- build and test with meson;
- doc/
Some of my changes are separate so you can see what I've done.
check_libpq_compression() is in the wrong place, but I couldn't
immediately see where else to put it, since src/common can't include the
backend's guc headers.
Some of the makefile changes seem unnecessary (now?), and my meson
changes don't seem quite right, either.
There's no reason for Zstd to be a separate patch anymore.
It should be updated to parse compression level and options using the
infrastructure introduced for basebackup.
And address the architectural issue from 2 years ago:
https://www.postgresql.org/message-id/20220118043919.GA23027%40telsasoft.com
The global variable PqStream should be moved into some libpq structure
(Port?) and handled within secure_read(). And pqsecure_read shouldn't
be passed as a function pointer/callback. And verify it passes tests
with all supported compression algorithms and connects to old servers.
--
Justin
Attachments: download all (4 files, .tar.gz)
0001-Implement-libpq-compression.patch (text/x-diff, 114.5K) download
From 6a74b2af0131499a3afa5132bc6dac08eb1884f6 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Fri, 14 Jan 2022 01:38:55 +0500
Subject: [PATCH 1/4] Implement libpq compression
---
doc/src/sgml/config.sgml | 21 +
doc/src/sgml/libpq.sgml | 37 +
doc/src/sgml/protocol.sgml | 158 +++
src/Makefile.global.in | 1 +
src/backend/Makefile | 8 +
src/backend/catalog/system_views.sql | 9 +
src/backend/libpq/pqcomm.c | 277 ++++-
src/backend/postmaster/postmaster.c | 10 +
.../libpqwalreceiver/libpqwalreceiver.c | 13 +-
src/backend/utils/activity/backend_status.c | 29 +
src/backend/utils/adt/pgstatfuncs.c | 50 +-
src/backend/utils/misc/guc_funcs.c | 21 +
src/backend/utils/misc/guc_tables.c | 10 +
src/bin/pgbench/pgbench.c | 17 +-
src/bin/psql/command.c | 17 +
src/common/Makefile | 4 +-
src/common/z_stream.c | 663 +++++++++++
src/common/zpq_stream.c | 1022 +++++++++++++++++
src/include/catalog/pg_proc.dat | 18 +-
src/include/common/z_stream.h | 109 ++
src/include/common/zpq_stream.h | 120 ++
src/include/libpq/libpq-be.h | 3 +
src/include/libpq/libpq.h | 1 +
src/include/libpq/pqcomm.h | 3 +
src/include/utils/backend_status.h | 7 +
src/interfaces/libpq/Makefile | 14 +
src/interfaces/libpq/exports.txt | 2 +
src/interfaces/libpq/fe-connect.c | 129 ++-
src/interfaces/libpq/fe-exec.c | 10 +-
src/interfaces/libpq/fe-misc.c | 92 +-
src/interfaces/libpq/fe-protocol3.c | 71 +-
src/interfaces/libpq/libpq-fe.h | 4 +
src/interfaces/libpq/libpq-int.h | 14 +
src/test/regress/expected/rules.out | 14 +-
src/tools/msvc/Mkvcbuild.pm | 2 +-
35 files changed, 2884 insertions(+), 96 deletions(-)
create mode 100644 src/common/z_stream.c
create mode 100644 src/common/zpq_stream.c
create mode 100644 src/include/common/z_stream.h
create mode 100644 src/include/common/zpq_stream.h
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bd50ea8e480..de4d9532cc5 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1045,6 +1045,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-libpq-compression" xreflabel="libpq_compression">
+ <term><varname>libpq_compression</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>libpq_compression</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter controls the available client-server traffic compression methods.
+ It allows rejecting compression requests even if it is supported by the server (for example, due to security, or CPU consumption).
+ The default is <literal>on</literal>, which means that all supported compression methods are allowed.
+ For more precise control, a list of the allowed compression methods can be specified.
+ For example, to allow only <literal>lz4</literal> and <literal>zlib</literal>, set the setting to <literal>lz4,zlib</literal>.
+ Also, maximal allowed compression level can be specified for each method, e.g. <literal>lz4:1,zlib:2</literal> setting will set the
+ maximal compression level for <literal>lz4</literal> to 1 and <literal>zlib</literal> to 2.
+ If a client requests the compression with a higher compression level, it will be set to the maximal allowed one.
+ Default (and recommended) maximal compression level for each algorithm is 1.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 3c9bd3d6730..d92f63ec350 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -1259,6 +1259,43 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-compression" xreflabel="compression">
+ <term><literal>compression</literal></term>
+ <listitem>
+ <para>
+ Request compression of libpq traffic. The client sends a request with
+ a list of compression algorithms. Compression can be requested by a client
+ by including the <literal>"compression"</literal> option in its connection string.
+ This can either be a boolean value to enable or disable compression
+ (<literal>"true"</literal>/<literal>"false"</literal>,
+ <literal>"on"</literal>/<literal>"off"</literal>,
+ <literal>"yes"</literal>/<literal>"no"</literal>,
+ <literal>"1"</literal>/<literal>"0"</literal>),
+ <literal>"any"</literal>,
+ or an explicit list of comma-separated compression algorithms
+ which can optionally include compression level (<literal>"zlib,lz4:2"</literal>).
+ If compression is enabled but an algorithm is not explicitly specified,
+ the client library sends its full list of supported algorithms.
+ The server intersects the received compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ </para>
+ <para>
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ Compressed data is transmitted via the CompressedData messages.
+ </para>
+ <para>
+ Support for compression algorithms must be enabled when the server is compiled.
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option). In both cases, streaming mode is used.
+ By default, compression is not requested by the client.
+ Please note that using compression together with SSL may expose extra vulnerabilities:
+ <ulink url="https://en.wikipedia.org/wiki/CRIME">CRIME</ulink>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-client-encoding" xreflabel="client_encoding">
<term><literal>client_encoding</literal></term>
<listitem>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 5fdd429e05d..147d55eab23 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -92,6 +92,15 @@
such as <command>COPY</command>.
</para>
+ <para>
+ It is possible to compress protocol data to reduce traffic and speed-up client-server interaction.
+ Compression is especially useful for importing/exporting data to/from the database using the <literal>COPY</literal> command
+ and for replication (both physical and logical). Compression can also reduce the server's response time
+ for queries returning a large amount of data (for example, JSON, BLOBs, text, ...).
+ Currently, two libraries are supported: zlib (default) and lz4 (if Postgres was
+ configured with --with-lz4 option).
+ </para>
+
<sect2 id="protocol-message-concepts">
<title>Messaging Overview</title>
@@ -262,6 +271,21 @@
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>CompressionAck</term>
+ <listitem>
+ <para>
+ The server accepts the client's compression request.
+ Compression is requested when a client connection includes the "compression" option, which contains a list of the requested compression algorithms.
+ The server intersects the requested compression algorithms with the allowed ones (controlled via the <literal>libpq_compression</literal> server config setting).
+ If the intersection is not empty, the server responds with CompressionAck containing the final list of the compression algorithms that can be used for the compression of libpq messages between the client and server.
+ If the intersection is empty (server does not accept any of the requested algorithms), then it replies with CompressionAck containing the empty list and it is up to the client whether to continue without compression or to report an error.
+ After sending the CompressionAck message, the server can send the SetCompressionMethod message to set the current compression algorithm for server-to-client traffic compression.
+ After receiving the CompressionAck message, the client can send the SetCompressionMethod message to set the current compression algorithm for client-to-server traffic compression.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term>AuthenticationOk</term>
<listitem>
@@ -4142,6 +4166,140 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+<varlistentry>
+<term>
+CompressionAck (B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('z')
+</term>
+<listitem>
+<para>
+ Acknowledge use of compression for protocol data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ List of the negotiated compression algorithms.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+
+<varlistentry>
+<term>
+SetCompressionMethod (F & B)
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('k')
+</term>
+<listitem>
+<para>
+ Switch the current compression algorithm. Following CompressedData messages will be compressed via the compressed algorithm specified in this message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32(5)
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ Index of the new compression algorithm.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+CompressedData (F & B)
+</term>
+<listitem>
+<para>
+
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('m')
+</term>
+<listitem>
+<para>
+ Identifies the message as compressed data.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32
+</term>
+<listitem>
+<para>
+ Length of message contents in bytes, including self.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Byte<replaceable>n</replaceable>
+</term>
+<listitem>
+<para>
+ Compressed message data.
+</para>
+</listitem>
+</varlistentry>
+</variablelist>
+
+</para>
+</listitem>
+</varlistentry>
+
<varlistentry id="protocol-message-formats-CopyData">
<term>CopyData (F & B)</term>
<listitem>
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index e96bedd4e7b..1b3dc97e97d 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -196,6 +196,7 @@ with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
+with_lz4 = @with_lz4@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 181c217fae4..f90e078b6ed 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -55,6 +55,14 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+endif
+
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2d8104b0907..89d18c12ef2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -871,6 +871,15 @@ CREATE VIEW pg_stat_activity AS
LEFT JOIN pg_database AS D ON (S.datid = D.oid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
+CREATE VIEW pg_stat_network_traffic AS
+ SELECT
+ S.pid,
+ S.rx_raw_bytes,
+ S.tx_raw_bytes,
+ S.rx_compressed_bytes,
+ S.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL) AS S;
+
CREATE VIEW pg_stat_replication AS
SELECT
S.pid,
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index ce56ab1d41d..9bf4672950c 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -60,6 +60,7 @@
#include <signal.h>
#include <fcntl.h>
#include <grp.h>
+#include <pgstat.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/socket.h>
@@ -75,11 +76,14 @@
#include "common/ip.h"
#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
#include "miscadmin.h"
#include "port/pg_bswap.h"
#include "storage/ipc.h"
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
+#include "utils/builtins.h"
+#include "common/zpq_stream.h"
/*
* Cope with the various platform-specific ways to spell TCP keepalive socket
@@ -106,6 +110,9 @@
int Unix_socket_permissions;
char *Unix_socket_group;
+/* GUC variable containing the allowed compression algorithms list (separated by comma) */
+char *libpq_compress_algorithms;
+
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -128,6 +135,9 @@ static char PqRecvBuffer[PQ_RECV_BUFFER_SIZE];
static int PqRecvPointer; /* Next index to read a byte from PqRecvBuffer */
static int PqRecvLength; /* End of data available in PqRecvBuffer */
+static ZpqStream * PqStream;
+
+
/*
* Message status
*/
@@ -163,6 +173,145 @@ const PQcommMethods *PqCommMethods = &PqCommSocketMethods;
WaitEventSet *FeBeWaitSet;
+static ssize_t
+write_compressed(void *arg, void const *data, size_t size)
+{
+ ssize_t rc = secure_write((Port *) arg, (void *) data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, 0, rc);
+ return rc;
+}
+
+static ssize_t
+read_compressed(void *arg, void *data, size_t size)
+{
+ ssize_t rc = secure_read((Port *) arg, data, size);
+
+ if (rc > 0)
+ pgstat_report_network_traffic(0, 0, rc, 0);
+ return rc;
+}
+
+/*
+ * Send the chosen compression algorithms to the client
+ */
+static void
+SendCompressionACK(char *compressors_str)
+{
+ StringInfoData buf;
+
+ pq_beginmessage(&buf, 'z');
+ pq_sendstring(&buf, compressors_str);
+ pq_endmessage(&buf);
+ pq_flush();
+}
+
+/* --------------------------------
+ * pq_configure - configure connection using port settings
+ *
+ * Right now only compression is toggled in the configure.
+ * Function returns 0 in case of success, non-null in case of error
+ * --------------------------------
+ */
+int
+pq_configure(Port *port)
+{
+ zpq_compressor *server_compressors;
+ zpq_compressor *client_compressors;
+ zpq_compressor *res_compressors;
+ size_t n_client_compressors;
+ size_t n_server_compressors;
+ size_t n_res_compressors = 0;
+ char *res_compressors_str;
+ char *client_compression_algorithms = port->compression_algorithms;
+
+ if (!client_compression_algorithms || !libpq_compress_algorithms)
+ {
+ return 0;
+ }
+
+ if (zpq_parse_compression_setting(libpq_compress_algorithms, &server_compressors, &n_server_compressors) == -1)
+ {
+ ereport(LOG, errmsg("failed to parse configured compression setting: %s", libpq_compress_algorithms));
+ return 0;
+ }
+
+ if (n_server_compressors == 0)
+ {
+ /*
+ * No enabled server compressors available, abort the compression
+ * initialization.
+ */
+ /* Send the compression acknowledgment with empty compressors list. */
+ char *empty_response = "";
+
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ if (!zpq_deserialize_compressors(client_compression_algorithms, &client_compressors, &n_client_compressors))
+ {
+ ereport(LOG, (errmsg("failed to parse received client compression methods: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ if (n_client_compressors == 0)
+ {
+ /*
+ * client did not provide any compression algorithms that server is
+ * compiled to support
+ */
+ ereport(LOG, (errmsg("server doesn't support any of the compression methods requested by the client: %s", client_compression_algorithms)));
+ return 0;
+ }
+
+ res_compressors = malloc(Max(n_client_compressors, n_server_compressors) * sizeof(zpq_compressor));
+
+ /*
+ * Intersect client and server compressors to determine the final list of
+ * the supported compressors. O(N^2) is negligible because of a small
+ * number of the compression methods.
+ */
+ for (size_t i = 0; i < n_client_compressors; i++)
+ {
+ for (size_t j = 0; j < n_server_compressors; j++)
+ {
+ if (client_compressors[i].impl == server_compressors[j].impl)
+ {
+ res_compressors[n_res_compressors].impl = client_compressors[i].impl;
+ /* prefer the lower compression level */
+ res_compressors[n_res_compressors++].level = Min(client_compressors[i].level, server_compressors[j].level);
+ break;
+ }
+ }
+ }
+ free(client_compressors);
+ free(server_compressors);
+
+ if (n_res_compressors == 0)
+ {
+ char *empty_response = "";
+
+ ereport(LOG, (errmsg("did not find any matches between the client and server compression methods, not enabling compression")));
+ free(res_compressors);
+ /* send the compression ack with empty compressors list */
+ SendCompressionACK(empty_response);
+ return 0;
+ }
+
+ res_compressors_str = zpq_serialize_compressors(res_compressors, n_res_compressors);
+ SendCompressionACK(res_compressors_str);
+
+ /* Init compression */
+ PqStream = zpq_create(res_compressors, n_res_compressors, write_compressed, read_compressed, MyProcPort, NULL, 0);
+ if (!PqStream)
+ {
+ ereport(LOG, (errmsg("failed to initialize the compression stream")));
+ return -1;
+ }
+ return 0;
+}
/* --------------------------------
* pq_init - initialize libpq at backend startup
@@ -266,6 +415,9 @@ socket_close(int code, Datum arg)
}
#endif /* ENABLE_GSS */
+ /* Release compression streams */
+ zpq_free(PqStream);
+
/*
* Cleanly shut down SSL layer. Nowhere else does a postmaster child
* call this, so this is safe when interrupting BackendInitialize().
@@ -902,12 +1054,15 @@ socket_set_nonblocking(bool nonblocking)
/* --------------------------------
* pq_recvbuf - load some bytes into the input buffer
*
- * returns 0 if OK, EOF if trouble
+ * nowait parameter toggles non-blocking mode.
+ * returns number of read bytes, EOF if trouble
* --------------------------------
*/
static int
-pq_recvbuf(void)
+pq_recvbuf(bool nowait)
{
+ int r;
+
if (PqRecvPointer > 0)
{
if (PqRecvLength > PqRecvPointer)
@@ -923,21 +1078,40 @@ pq_recvbuf(void)
}
/* Ensure that we're in blocking mode */
- socket_set_nonblocking(false);
+ socket_set_nonblocking(nowait);
/* Can fill buffer from PqRecvLength and upwards */
for (;;)
{
- int r;
-
- r = secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
- PQ_RECV_BUFFER_SIZE - PqRecvLength);
+ /*
+ * If streaming compression is enabled then use correspondent
+ * compression read function.
+ */
+ r = PqStream
+ ? zpq_read(PqStream, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength, false)
+ : secure_read(MyProcPort, PqRecvBuffer + PqRecvLength,
+ PQ_RECV_BUFFER_SIZE - PqRecvLength);
if (r < 0)
{
+ if (r == ZS_DECOMPRESS_ERROR)
+ {
+ char const *msg = zpq_decompress_error(PqStream);
+
+ if (msg == NULL)
+ msg = "end of stream";
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to decompress data: %s", msg)));
+ return EOF;
+ }
if (errno == EINTR)
continue; /* Ok if interrupted */
+ if (nowait && (errno == EAGAIN || errno == EWOULDBLOCK))
+ return 0;
+
/*
* Careful: an ereport() that tries to write to the client would
* cause recursion to here, leading to stack overflow and core
@@ -958,7 +1132,8 @@ pq_recvbuf(void)
}
/* r contains number of bytes read, so just incr length */
PqRecvLength += r;
- return 0;
+ pgstat_report_network_traffic(r, 0, 0, 0);
+ return r;
}
}
@@ -973,7 +1148,8 @@ pq_getbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer++];
@@ -992,7 +1168,8 @@ pq_peekbyte(void)
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
return (unsigned char) PqRecvBuffer[PqRecvPointer];
@@ -1009,48 +1186,15 @@ pq_peekbyte(void)
int
pq_getbyte_if_available(unsigned char *c)
{
- int r;
+ int r = 0;
Assert(PqCommReadingMsg);
- if (PqRecvPointer < PqRecvLength)
+ if (PqRecvPointer < PqRecvLength || (r = pq_recvbuf(true)) > 0)
{
*c = PqRecvBuffer[PqRecvPointer++];
return 1;
}
-
- /* Put the socket into non-blocking mode */
- socket_set_nonblocking(true);
-
- r = secure_read(MyProcPort, c, 1);
- if (r < 0)
- {
- /*
- * Ok if no data available without blocking or interrupted (though
- * EINTR really shouldn't happen with a non-blocking socket). Report
- * other errors.
- */
- if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
- r = 0;
- else
- {
- /*
- * Careful: an ereport() that tries to write to the client would
- * cause recursion to here, leading to stack overflow and core
- * dump! This message must go *only* to the postmaster log.
- */
- ereport(COMMERROR,
- (errcode_for_socket_access(),
- errmsg("could not receive data from client: %m")));
- r = EOF;
- }
- }
- else if (r == 0)
- {
- /* EOF detected */
- r = EOF;
- }
-
return r;
}
@@ -1071,7 +1215,8 @@ pq_getbytes(char *s, size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1105,7 +1250,8 @@ pq_discardbytes(size_t len)
{
while (PqRecvPointer >= PqRecvLength)
{
- if (pq_recvbuf()) /* If nothing in buffer, then recv some */
+ if (pq_recvbuf(false) == EOF) /* If nothing in buffer, then recv
+ * some */
return EOF; /* Failed to recv data */
}
amount = PqRecvLength - PqRecvPointer;
@@ -1129,7 +1275,6 @@ pq_buffer_has_data(void)
return (PqRecvPointer < PqRecvLength);
}
-
/* --------------------------------
* pq_startmsgread - begin reading a message from the client.
*
@@ -1333,13 +1478,24 @@ internal_flush(void)
char *bufptr = PqSendBuffer + PqSendStart;
char *bufend = PqSendBuffer + PqSendPointer;
- while (bufptr < bufend)
+ while (bufptr < bufend || zpq_buffered_tx(PqStream) != 0)
+
+ /*
+ * has more data to flush or unsent data in internal compression
+ * buffer
+ */
{
int r;
+ size_t processed = 0;
+ size_t available = bufend - bufptr;
- r = secure_write(MyProcPort, bufptr, bufend - bufptr);
+ r = PqStream
+ ? zpq_write(PqStream, bufptr, available, &processed)
+ : secure_write(MyProcPort, bufptr, available);
+ bufptr += processed;
+ PqSendStart += processed;
- if (r <= 0)
+ if (r < 0 || (r == 0 && available))
{
if (errno == EINTR)
continue; /* Ok if we were interrupted */
@@ -1382,12 +1538,12 @@ internal_flush(void)
InterruptPending = 1;
return EOF;
}
+ pgstat_report_network_traffic(0, r, 0, 0);
last_reported_send_errno = 0; /* reset after any successful send */
bufptr += r;
PqSendStart += r;
}
-
PqSendStart = PqSendPointer = 0;
return 0;
}
@@ -1404,7 +1560,7 @@ socket_flush_if_writable(void)
int res;
/* Quick exit if nothing to do */
- if (PqSendPointer == PqSendStart)
+ if ((PqSendPointer == PqSendStart) && (zpq_buffered_tx(PqStream) == 0))
return 0;
/* No-op if reentrant call */
@@ -1427,7 +1583,7 @@ socket_flush_if_writable(void)
static bool
socket_is_send_pending(void)
{
- return (PqSendStart < PqSendPointer);
+ return (PqSendStart < PqSendPointer || (zpq_buffered_tx(PqStream) != 0));
}
/* --------------------------------
@@ -2052,3 +2208,16 @@ retry:
return true;
}
+
+PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+
+Datum
+pg_compression_algorithm(PG_FUNCTION_ARGS)
+{
+ char const *algorithm_name = PqStream ? zpq_compress_algorithm_name(PqStream) : NULL;
+
+ if (algorithm_name)
+ PG_RETURN_TEXT_P(cstring_to_text(algorithm_name));
+ else
+ PG_RETURN_NULL();
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 1da5752047f..7d747755368 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -2230,6 +2230,8 @@ retry1:
port->database_name = pstrdup(valptr);
else if (strcmp(nameptr, "user") == 0)
port->user_name = pstrdup(valptr);
+ else if (strcmp(nameptr, "_pq_.compression") == 0)
+ port->compression_algorithms = pstrdup(valptr);
else if (strcmp(nameptr, "options") == 0)
port->cmdline_options = pstrdup(valptr);
else if (strcmp(nameptr, "replication") == 0)
@@ -4406,6 +4408,14 @@ BackendInitialize(Port *port)
if (status != STATUS_OK)
proc_exit(0);
+ if (pq_configure(port))
+ {
+ ereport(COMMERROR,
+ (errcode_for_socket_access(),
+ errmsg("failed to send compression message: %m")));
+ proc_exit(0);
+ }
+
/*
* Now that we have the user and database name, we can set the process
* title for ps. It's good to do this as early as possible in startup.
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 7f697b0f292..6584105ed77 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -201,11 +201,14 @@ libpqrcv_connect(const char *conninfo, bool logical, const char *appname,
else
io_flag = WL_SOCKET_WRITEABLE;
- rc = WaitLatchOrSocket(MyLatch,
- WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
- PQsocket(conn->streamConn),
- 0,
- WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
+ if (status == PGRES_POLLING_READING && PQreadPending(conn->streamConn))
+ rc = WL_SOCKET_READABLE;
+ else
+ rc = WaitLatchOrSocket(MyLatch,
+ WL_EXIT_ON_PM_DEATH | WL_LATCH_SET | io_flag,
+ PQsocket(conn->streamConn),
+ 0,
+ WAIT_EVENT_LIBPQWALRECEIVER_CONNECT);
/* Interrupted? */
if (rc & WL_LATCH_SET)
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 1146a6c33cd..7cdd4acbb13 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -338,6 +338,9 @@ pgstat_bestart(void)
lbeentry.st_xact_start_timestamp = 0;
lbeentry.st_databaseid = MyDatabaseId;
+ lbeentry.st_tx_raw_bytes = lbeentry.st_rx_raw_bytes =
+ lbeentry.st_tx_compressed_bytes = lbeentry.st_rx_compressed_bytes = 0;
+
/* We have userid for client-backends, wal-sender and bgworker processes */
if (lbeentry.st_backendType == B_BACKEND
|| lbeentry.st_backendType == B_WAL_SENDER
@@ -716,6 +719,32 @@ pgstat_report_xact_timestamp(TimestampTz tstamp)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Report network raw or compressed tx/rx traffic as the specified values.
+ */
+void
+pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+
+ if (!pgstat_track_activities || !beentry)
+ return;
+
+ /*
+ * Update my status entry, following the protocol of bumping
+ * st_changecount before and after. We use a volatile pointer here to
+ * ensure the compiler doesn't try to get cute.
+ */
+ PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
+
+ beentry->st_rx_raw_bytes += rx_raw_bytes;
+ beentry->st_tx_raw_bytes += tx_raw_bytes;
+ beentry->st_rx_compressed_bytes += rx_compressed_bytes;
+ beentry->st_tx_compressed_bytes += tx_compressed_bytes;
+
+ PGSTAT_END_WRITE_ACTIVITY(beentry);
+}
+
/* ----------
* pgstat_read_current_status() -
*
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index ae3365d9171..9ce5529f23d 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -559,7 +559,7 @@ pg_stat_get_progress_info(PG_FUNCTION_ARGS)
Datum
pg_stat_get_activity(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_ACTIVITY_COLS 30
+#define PG_STAT_GET_ACTIVITY_COLS 34
int num_backends = pgstat_fetch_stat_numbackends();
int curr_backend;
int pid = PG_ARGISNULL(0) ? -1 : PG_GETARG_INT32(0);
@@ -858,6 +858,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[29] = true;
else
values[29] = UInt64GetDatum(beentry->st_query_id);
+ values[30] = beentry->st_rx_raw_bytes;
+ values[31] = beentry->st_tx_raw_bytes;
+ values[32] = beentry->st_rx_compressed_bytes;
+ values[33] = beentry->st_tx_compressed_bytes;
}
else
{
@@ -886,6 +890,10 @@ pg_stat_get_activity(PG_FUNCTION_ARGS)
nulls[27] = true;
nulls[28] = true;
nulls[29] = true;
+ nulls[30] = true;
+ nulls[31] = true;
+ nulls[32] = true;
+ nulls[33] = true;
}
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
@@ -1082,6 +1090,46 @@ pg_stat_get_backend_start(PG_FUNCTION_ARGS)
PG_RETURN_TIMESTAMPTZ(result);
}
+Datum
+pg_stat_get_network_traffic(PG_FUNCTION_ARGS)
+{
+#define PG_STAT_NETWORK_TRAFFIC_COLS 4
+ TupleDesc tupdesc;
+ Datum values[PG_STAT_NETWORK_TRAFFIC_COLS];
+ bool nulls[PG_STAT_NETWORK_TRAFFIC_COLS];
+ int32 beid = PG_GETARG_INT32(0);
+ PgBackendStatus *beentry;
+
+ if ((beentry = pgstat_fetch_stat_beentry(beid)) == NULL)
+ PG_RETURN_NULL();
+ else if (!HAS_PGSTAT_PERMISSIONS(beentry->st_userid))
+ PG_RETURN_NULL();
+
+ /* Initialise values and NULL flags arrays */
+ MemSet(values, 0, sizeof(values));
+ MemSet(nulls, 0, sizeof(nulls));
+
+ /* Initialise attributes information in the tuple descriptor */
+ tupdesc = CreateTemplateTupleDesc(PG_STAT_NETWORK_TRAFFIC_COLS);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 1, "rx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 2, "tx_raw_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 3, "rx_compressed_bytes",
+ INT8OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 4, "tx_compressed_bytes",
+ INT8OID, -1, 0);
+ BlessTupleDesc(tupdesc);
+
+ /* Fill values and NULLs */
+ values[0] = beentry->st_rx_raw_bytes;
+ values[1] = beentry->st_tx_raw_bytes;
+ values[2] = beentry->st_rx_compressed_bytes;
+ values[3] = beentry->st_tx_compressed_bytes;
+
+ /* Returns the record as Datum */
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
Datum
pg_stat_get_backend_client_addr(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 108b3bd1290..5585f992702 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -1049,3 +1049,24 @@ show_all_file_settings(PG_FUNCTION_ARGS)
return (Datum) 0;
}
+
+
+#include "common/zpq_stream.h"
+static bool check_libpq_compression(char **newval, void **extra, GucSource source);
+
+static bool
+check_libpq_compression(char **newval, void **extra, GucSource source)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(*newval, &compressors, &n_compressors))
+ {
+ GUC_check_errdetail("Cannot parse the libpq_compression setting.");
+ return false;
+ }
+
+ free(compressors);
+ return true;
+}
+
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 836b49484a1..bef045e5be9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4447,6 +4447,16 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"libpq_compression", PGC_SIGHUP, CLIENT_CONN_OTHER,
+ gettext_noop("Sets the list of allowed libpq compression algorithms."),
+ NULL
+ },
+ &libpq_compress_algorithms,
+ "off",
+ check_libpq_compression, NULL, NULL
+ },
+
{
{"backtrace_functions", PGC_SUSET, DEVELOPER_OPTIONS,
gettext_noop("Log backtrace for errors in these functions."),
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 36905a89681..6b7c2366cf9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -7355,6 +7355,9 @@ threadRun(void *arg)
int nsocks; /* number of sockets to be waited for */
pg_time_usec_t min_usec;
pg_time_usec_t now = 0; /* set this only if needed */
+ bool buffered_rx = false; /* true if some of the clients has
+ * data left in SSL/ZPQ read
+ * buffers */
/*
* identify which client sockets should be checked for input, and
@@ -7390,6 +7393,9 @@ threadRun(void *arg)
*/
int sock = PQsocket(st->con);
+ /* check if conn has buffered SSL / ZPQ read data */
+ buffered_rx = buffered_rx || PQreadPending(st->con);
+
if (sock < 0)
{
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
@@ -7433,7 +7439,7 @@ threadRun(void *arg)
{
if (nsocks > 0)
{
- rc = wait_on_socket_set(sockets, min_usec);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, min_usec);
}
else /* nothing active, simple sleep */
{
@@ -7442,7 +7448,7 @@ threadRun(void *arg)
}
else /* no explicit delay, wait without timeout */
{
- rc = wait_on_socket_set(sockets, 0);
+ rc = buffered_rx ? 1 : wait_on_socket_set(sockets, 0);
}
if (rc < 0)
@@ -7482,8 +7488,11 @@ threadRun(void *arg)
pg_log_error("invalid socket: %s", PQerrorMessage(st->con));
goto done;
}
-
- if (!socket_has_input(sockets, sock, nsocks++))
+ if (PQreadPending(st->con))
+ {
+ nsocks++;
+ }
+ else if (!socket_has_input(sockets, sock, nsocks++))
continue;
}
else if (st->state == CSTATE_FINISHED ||
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index 7672ed9e9d5..64c0f4cce88 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -172,6 +172,7 @@ static void print_with_linenumbers(FILE *output, char *lines,
const char *header_keyword);
static void minimal_error_message(PGresult *res);
+static void printCompressionInfo(void);
static void printSSLInfo(void);
static void printGSSInfo(void);
static bool printPsetInfo(const char *param, printQueryOpt *popt);
@@ -676,6 +677,7 @@ exec_command_conninfo(PsqlScanState scan_state, bool active_branch)
printf(_("You are connected to database \"%s\" as user \"%s\" on host \"%s\" at port \"%s\".\n"),
db, PQuser(pset.db), host, PQport(pset.db));
}
+ printCompressionInfo();
printSSLInfo();
printGSSInfo();
}
@@ -3693,6 +3695,21 @@ connection_warnings(bool in_startup)
}
}
+/*
+ * printCompressionInfo
+ *
+ * Print information about used compressor/decompressor
+ */
+static void
+printCompressionInfo(void)
+{
+ char *algorithms = PQcompression (pset.db);
+
+ if (algorithms != NULL)
+ {
+ printf(_("Compression: %s\n"), algorithms);
+ }
+}
/*
* printSSLInfo
diff --git a/src/common/Makefile b/src/common/Makefile
index e9af7346c9c..d200f5feb58 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -79,7 +79,9 @@ OBJS_COMMON = \
unicode_norm.o \
username.o \
wait_error.o \
- wchar.o
+ wchar.o \
+ z_stream.o \
+ zpq_stream.o
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
new file mode 100644
index 00000000000..b75f194a1cf
--- /dev/null
+++ b/src/common/z_stream.c
@@ -0,0 +1,663 @@
+#include "c.h"
+#include "pg_config.h"
+#include "common/z_stream.h"
+
+/*
+ * Functions implementing streaming compression algorithm
+ */
+typedef struct
+{
+ /*
+ * Name of compression algorithm.
+ */
+ char const *(*name) (void);
+
+ /*
+ * Create new compression stream. level: compression level
+ */
+ void *(*create_compressor) (int level);
+
+ /*
+ * Create new decompression stream.
+ */
+ void *(*create_decompressor) ();
+
+ /*
+ * Decompress up to "src_size" compressed bytes from *src and write up to
+ * "dst_size" raw (decompressed) bytes to *dst. Number of decompressed
+ * bytes written to *dst is stored in *dst_processed. Number of compressed
+ * bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during decompression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * decompressor internal buffers.
+ *
+ * ZS_STREAM_END if encountered end of compressed data stream.
+ *
+ * ZS_DECOMPRESS_ERROR if encountered an error during decompression
+ * attempt.
+ */
+ ssize_t (*decompress) (void *ds, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Compress up to "src_size" raw (non-compressed) bytes from *src and
+ * write up to "dst_size" compressed bytes to *dst. Number of compressed
+ * bytes written to *dst is stored in *dst_processed. Number of
+ * non-compressed bytes read from *src is stored in *src_processed.
+ *
+ * Return codes: ZS_OK if no errors were encountered during compression
+ * attempt. This return code does not guarantee that *src_processed > 0 or
+ * *dst_processed > 0.
+ *
+ * ZS_DATA_PENDING means that there might be some data left within
+ * compressor internal buffers.
+ *
+ * ZS_COMPRESS_ERROR if encountered an error during compression attempt.
+ */
+ ssize_t (*compress) (void *cs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+ /*
+ * Free compression stream created by create_compressor function.
+ */
+ void (*free_compressor) (void *cs);
+
+ /*
+ * Free decompression stream created by create_decompressor function.
+ */
+ void (*free_decompressor) (void *ds);
+
+ /*
+ * Get compressor error message.
+ */
+ char const *(*compress_error) (void *cs);
+
+ /*
+ * Get decompressor error message.
+ */
+ char const *(*decompress_error) (void *ds);
+
+ ssize_t (*end_compression) (void *cs, void *dst, size_t dst_size, size_t *dst_processed);
+} ZAlgorithm;
+
+struct ZStream
+{
+ ZAlgorithm const *algorithm;
+ void *stream;
+ bool not_flushed;
+};
+
+#if HAVE_LIBZ
+
+#include <stdlib.h>
+#include <zlib.h>
+
+
+static void *
+zlib_create_compressor(int level)
+{
+ int rc;
+ z_stream *c_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(c_stream, 0, sizeof(*c_stream));
+ rc = deflateInit(c_stream, level);
+ if (rc != Z_OK)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+zlib_create_decompressor()
+{
+ int rc;
+ z_stream *d_stream = (z_stream *) malloc(sizeof(z_stream));
+
+ memset(d_stream, 0, sizeof(*d_stream));
+ rc = inflateInit(d_stream);
+ if (rc != Z_OK)
+ {
+ free(d_stream);
+ return NULL;
+ }
+ return d_stream;
+}
+
+static ssize_t
+zlib_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *ds = (z_stream *) d_stream;
+ int rc;
+
+ ds->next_in = (Bytef *) src;
+ ds->avail_in = src_size;
+ ds->next_out = (Bytef *) dst;
+ ds->avail_out = dst_size;
+
+ rc = inflate(ds, Z_SYNC_FLUSH);
+ *src_processed = src_size - ds->avail_in;
+ *dst_processed = dst_size - ds->avail_out;
+
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_STREAM_END;
+ }
+ if (rc != Z_OK && rc != Z_BUF_ERROR)
+ {
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zlib_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc PG_USED_FOR_ASSERTS_ONLY;
+ unsigned deflate_pending = 0;
+
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = (Bytef *) src;
+ cs->avail_in = src_size;
+
+ rc = deflate(cs, Z_SYNC_FLUSH);
+ Assert(rc == Z_OK);
+ *dst_processed = dst_size - cs->avail_out;
+ *src_processed = src_size - cs->avail_in;
+
+ deflatePending(cs, &deflate_pending, Z_NULL); /* check if any data left
+ * in deflate buffer */
+ if (deflate_pending > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+zlib_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ z_stream *cs = (z_stream *) c_stream;
+ int rc;
+
+ cs->next_out = (Bytef *) dst;
+ cs->avail_out = dst_size;
+ cs->next_in = NULL;
+ cs->avail_in = 0;
+
+ rc = deflate(cs, Z_STREAM_END);
+ Assert(rc == Z_OK || rc == Z_STREAM_END);
+ *dst_processed = dst_size - cs->avail_out;
+ if (rc == Z_STREAM_END)
+ {
+ return ZS_OK;
+ }
+
+ return ZS_DATA_PENDING;
+}
+
+static void
+zlib_free_compressor(void *c_stream)
+{
+ z_stream *cs = (z_stream *) c_stream;
+
+ if (cs != NULL)
+ {
+ deflateEnd(cs);
+ free(cs);
+ }
+}
+
+static void
+zlib_free_decompressor(void *d_stream)
+{
+ z_stream *ds = (z_stream *) d_stream;
+
+ if (ds != NULL)
+ {
+ inflateEnd(ds);
+ free(ds);
+ }
+}
+
+static char const *
+zlib_error(void *stream)
+{
+ z_stream *zs = (z_stream *) stream;
+
+ return zs->msg;
+}
+
+static char const *
+zlib_name(void)
+{
+ return "zlib";
+}
+
+#endif
+
+#if USE_LZ4
+#include <lz4.h>
+
+#define MESSAGE_MAX_BYTES 819200
+#define RING_BUFFER_BYTES (1024 * 64 + MESSAGE_MAX_BYTES)
+
+typedef struct ZS_LZ4_CStream
+{
+ LZ4_stream_t *stream;
+ int level;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_CStream;
+
+typedef struct ZS_LZ4_DStream
+{
+ LZ4_streamDecode_t *stream;
+ size_t buf_pos;
+ char buf[RING_BUFFER_BYTES];
+} ZS_LZ4_DStream;
+
+static void *
+lz4_create_compressor(int level)
+{
+ ZS_LZ4_CStream *c_stream = (ZS_LZ4_CStream *) malloc(sizeof(ZS_LZ4_CStream));
+
+ if (c_stream == NULL)
+ {
+ return NULL;
+ }
+ c_stream->stream = LZ4_createStream();
+ c_stream->level = level;
+ c_stream->buf_pos = 0;
+ if (c_stream->stream == NULL)
+ {
+ free(c_stream);
+ return NULL;
+ }
+ return c_stream;
+}
+
+static void *
+lz4_create_decompressor()
+{
+ ZS_LZ4_DStream *d_stream = (ZS_LZ4_DStream *) malloc(sizeof(ZS_LZ4_DStream));
+
+ if (d_stream == NULL)
+ {
+ return NULL;
+ }
+
+ d_stream->stream = LZ4_createStreamDecode();
+ d_stream->buf_pos = 0;
+ if (d_stream->stream == NULL)
+ {
+ free(d_stream);
+ return NULL;
+ }
+
+ return d_stream;
+}
+
+static ssize_t
+lz4_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ char *const decPtr = &ds->buf[ds->buf_pos];
+
+ const int decBytes = LZ4_decompress_safe_continue(
+ ds->stream, src, decPtr, (int) src_size, (int) dst_size);
+ Assert(decBytes > 0);
+
+ *dst_processed = decBytes;
+ *src_processed = src_size;
+
+ memcpy(dst, decPtr, decBytes); /* write msg length */
+
+ ds->buf_pos += decBytes;
+ if (ds->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ ds->buf_pos = 0;
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+lz4_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+ int cmpBytes;
+
+ src_size = Min(MESSAGE_MAX_BYTES, src_size);
+
+ memcpy((char *) (cs->buf) + cs->buf_pos, src, src_size); /* write msg length */
+
+ Assert(dst_size >= LZ4_compressBound(src_size));
+
+ cmpBytes = LZ4_compress_fast_continue(
+ cs->stream, (char *) (cs->buf) + cs->buf_pos, dst, (int) src_size, (int) dst_size, cs->level);
+
+ Assert(cmpBytes > 0);
+ Assert(cmpBytes <= MESSAGE_MAX_BYTES);
+
+ *dst_processed = cmpBytes;
+ *src_processed = src_size;
+
+ cs->buf_pos += src_size;
+ if (cs->buf_pos >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
+ {
+ cs->buf_pos = 0;
+ }
+ return ZS_OK;
+}
+
+
+static ssize_t
+lz4_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ *dst_processed = 0;
+ return ZS_OK;
+}
+
+static void
+lz4_free_compressor(void *c_stream)
+{
+ ZS_LZ4_CStream *cs = (ZS_LZ4_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ if (cs->stream != NULL)
+ {
+ LZ4_freeStream(cs->stream);
+ }
+ free(cs);
+ }
+}
+
+static void
+lz4_free_decompressor(void *d_stream)
+{
+ ZS_LZ4_DStream *ds = (ZS_LZ4_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ if (ds->stream != NULL)
+ {
+ LZ4_freeStreamDecode(ds->stream);
+ }
+ free(ds);
+ }
+}
+
+static char const *
+lz4_error(void *stream)
+{
+ /* lz4 doesn't have any explicit API to get the error names */
+ return "NO_MSG";
+}
+
+static char const *
+lz4_name(void)
+{
+ return "lz4";
+}
+
+#endif
+
+static char const *
+no_compression_name(void)
+{
+ return NULL;
+}
+
+/*
+ * Array with all supported compression algorithms.
+ */
+static ZAlgorithm const zs_algorithms[] =
+{
+#if HAVE_LIBZ
+ {zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
+#endif
+#if USE_LZ4
+ {lz4_name, lz4_create_compressor, lz4_create_decompressor, lz4_decompress, lz4_compress, lz4_free_compressor, lz4_free_decompressor, lz4_error, lz4_error, lz4_end},
+#endif
+ {no_compression_name}
+};
+
+inline bool
+zs_is_valid_impl_id(unsigned int id)
+{
+ return id >= 0 && id < lengthof(zs_algorithms);
+}
+
+static ssize_t
+zs_init_compressor(ZStream * zs, unsigned int c_alg_impl, int c_level)
+{
+ if (!zs_is_valid_impl_id(c_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[c_alg_impl];
+ zs->stream = zs->algorithm->create_compressor(c_level);
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+static ssize_t
+zs_init_decompressor(ZStream * zs, unsigned int d_alg_impl)
+{
+ if (!zs_is_valid_impl_id(d_alg_impl))
+ {
+ return -1;
+ }
+ zs->algorithm = &zs_algorithms[d_alg_impl];
+ zs->stream = zs->algorithm->create_decompressor();
+ if (zs->stream == NULL)
+ {
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Index of used compression algorithm in zs_algorithms array.
+ */
+ZStream *
+zs_create_compressor(unsigned int c_alg_impl, int c_level)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_compressor(zs, c_alg_impl, c_level))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ZStream *
+zs_create_decompressor(unsigned int d_alg_impl)
+{
+ ZStream *zs = (ZStream *) malloc(sizeof(ZStream));
+
+ zs->not_flushed = false;
+
+ if (zs_init_decompressor(zs, d_alg_impl))
+ {
+ free(zs);
+ return NULL;
+ }
+
+ return zs;
+}
+
+ssize_t
+zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *src_processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->decompress(zs->stream,
+ src, src_size, src_processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+
+ if (rc == ZS_OK || rc == ZS_INCOMPLETE_SRC || rc == ZS_STREAM_END)
+ {
+ return rc;
+ }
+
+ return ZS_DECOMPRESS_ERROR;
+}
+
+ssize_t
+zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *processed = 0;
+ *dst_processed = 0;
+
+ rc = zs->algorithm->compress(zs->stream,
+ buf, size, processed,
+ dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+void
+zs_compressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_compressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+void
+zs_decompressor_free(ZStream * zs)
+{
+ if (zs == NULL)
+ {
+ return;
+ }
+
+ if (zs->stream)
+ {
+ zs->algorithm->free_decompressor(zs->stream);
+ }
+
+ free(zs);
+}
+
+ssize_t
+zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ssize_t rc;
+
+ *dst_processed = 0;
+
+ rc = zs->algorithm->end_compression(zs->stream, dst, dst_size, dst_processed);
+
+ zs->not_flushed = false;
+ if (rc == ZS_DATA_PENDING)
+ {
+ zs->not_flushed = true;
+ return ZS_OK;
+ }
+ if (rc != ZS_OK)
+ {
+ return ZS_COMPRESS_ERROR;
+ }
+
+ return rc;
+}
+
+char const *
+zs_compress_error(ZStream * zs)
+{
+ return zs->algorithm->compress_error(zs->stream);
+}
+
+char const *
+zs_decompress_error(ZStream * zs)
+{
+ return zs->algorithm->decompress_error(zs->stream);
+}
+
+bool
+zs_buffered(ZStream * zs)
+{
+ return zs ? zs->not_flushed : 0;
+}
+
+
+/*
+ * Get list of the supported algorithms.
+ */
+char **
+zs_get_supported_algorithms(void)
+{
+ size_t n_algorithms = lengthof(zs_algorithms);
+ char **algorithm_names = malloc(n_algorithms * sizeof(char *));
+
+ for (size_t i = 0; i < n_algorithms; i++)
+ {
+ algorithm_names[i] = (char *) zs_algorithms[i].name();
+ }
+
+ return algorithm_names;
+}
+
+char const *
+zs_compress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
+
+char const *
+zs_decompress_algorithm_name(ZStream * zs)
+{
+ return zs ? zs->algorithm->name() : NULL;
+}
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
new file mode 100644
index 00000000000..a712200e453
--- /dev/null
+++ b/src/common/zpq_stream.c
@@ -0,0 +1,1022 @@
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+#include <unistd.h>
+#include <math.h>
+
+#include "common/zpq_stream.h"
+#include "pg_config.h"
+#include "port/pg_bswap.h"
+
+/* log warnings on backend */
+#ifndef FRONTEND
+#define pg_log_warning(...) elog(WARNING, __VA_ARGS__)
+#else
+#define pg_log_warning(...) (void)0
+#endif
+
+/* ZpqBuffer size, in bytes */
+#define ZPQ_BUFFER_SIZE 8192000
+/* CompressedData msg_type */
+#define ZPQ_COMPRESSED_MSG_TYPE 'm'
+/* SetCompressionMethod msg_type */
+#define ZPQ_SET_COMPRESSION_MSG_TYPE 'k'
+
+#define ZPQ_COMPRESS_THRESHOLD 60
+
+typedef struct ZpqBuffer ZpqBuffer;
+
+
+/* ZpqBuffer used as RX/TX buffer in ZpqStream */
+struct ZpqBuffer
+{
+ char buf[ZPQ_BUFFER_SIZE];
+ size_t size; /* current size of buf */
+ size_t pos; /* current position in buf, in range [0, size] */
+};
+
+static inline void
+zpq_buf_init(ZpqBuffer * zb)
+{
+ zb->size = 0;
+ zb->pos = 0;
+}
+
+static inline size_t
+zpq_buf_left(ZpqBuffer * zb)
+{
+ Assert(zb->buf);
+ return ZPQ_BUFFER_SIZE - zb->size;
+}
+
+static inline size_t
+zpq_buf_unread(ZpqBuffer * zb)
+{
+ return zb->size - zb->pos;
+}
+
+static inline char *
+zpq_buf_size(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->size;
+}
+
+static inline char *
+zpq_buf_pos(ZpqBuffer * zb)
+{
+ return (char *) (zb->buf) + zb->pos;
+}
+
+static inline void
+zpq_buf_size_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->size += value;
+}
+
+static inline void
+zpq_buf_pos_advance(ZpqBuffer * zb, size_t value)
+{
+ zb->pos += value;
+}
+
+static inline void
+zpq_buf_reuse(ZpqBuffer * zb)
+{
+ size_t unread = zpq_buf_unread(zb);
+
+ if (unread > 5) /* can read message header, don't do anything */
+ return;
+ if (unread == 0)
+ {
+ zb->size = 0;
+ zb->pos = 0;
+ return;
+ }
+ memmove(zb->buf, zb->buf + zb->pos, unread);
+ zb->size = unread;
+ zb->pos = 0;
+}
+
+struct ZpqStream
+{
+ ZStream *c_stream; /* underlying compression stream */
+ ZStream *d_stream; /* underlying decompression stream */
+
+ size_t tx_total; /* amount of bytes sent to tx_func */
+
+ size_t tx_total_raw; /* amount of bytes received by zpq_write */
+ size_t rx_total; /* amount of bytes read by rx_func */
+ size_t rx_total_raw; /* amount of bytes returned by zpq_write */
+ bool is_compressing; /* current compression state */
+
+ bool is_decompressing; /* current decompression state */
+ size_t rx_msg_bytes_left; /* number of bytes left to process without
+ * changing the decompression state */
+ size_t tx_msg_bytes_left; /* number of bytes left to process without
+ * changing the compression state */
+
+ ZpqBuffer rx_in; /* buffer for unprocessed data read by rx_func */
+ ZpqBuffer tx_in; /* buffer for unprocessed data consumed by
+ * zpq_write */
+ ZpqBuffer tx_out; /* buffer for processed data waiting for send
+ * via tx_func */
+
+ zpq_rx_func rx_func;
+ zpq_tx_func tx_func;
+ void *arg;
+
+ zpq_compressor *compressors; /* compressors array holds the available
+ * compressors to use for
+ * compression/decompression */
+ size_t n_compressors; /* size of the compressors array */
+ int compress_alg_idx; /* index of the active compression
+ * algorithm */
+ int decompress_alg_idx; /* index of the active decompression
+ * algorithm */
+ int compressor_by_msg_type[256]; /* map to choose a compressor
+ * by the protocol message
+ * type */
+
+ bool reading_set_compression; /* utility marker indicating
+ * partial SetCompressionMethod
+ * read */
+};
+
+/*
+ * Message compression map defines the logic for choosing the compressor
+ * based on the protocol message type. Currently, it is a basic prototype to demonstrate
+ * the capabilities of the on-the-fly compression switch.
+ */
+static inline void
+zpq_build_msg_compression_map(ZpqStream * zpq)
+{
+ int i;
+
+ for (i = 0; i < 256; i++)
+ {
+ zpq->compressor_by_msg_type[i] = -1;
+ }
+
+ for (i = 0; i < zpq->n_compressors; i++)
+ {
+ /* compress CopyData, DataRow and Query messages */
+ if (zpq->compressor_by_msg_type['d'] == -1)
+ {
+ zpq->compressor_by_msg_type['d'] = i;
+ }
+ if (zpq->compressor_by_msg_type['D'] == -1)
+ {
+ zpq->compressor_by_msg_type['D'] = i;
+ }
+ if (zpq->compressor_by_msg_type['Q'] == -1)
+ {
+ zpq->compressor_by_msg_type['Q'] = i;
+ }
+ }
+}
+
+/*
+ * Choose the index of compressor to use for the message of msg_type with msg_len.
+ * Return values:
+ * - the non-negative index of zpq->compressors array
+ * - -1, if message should not be compressed
+ */
+static inline int
+zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
+ {
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
+ }
+ return -1;
+}
+
+/*
+ * Check if should compress message of msg_type with msg_len.
+ * Return true if should, false if should not.
+ */
+static inline bool
+zpq_should_compress(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ return zpq_choose_compressor(zpq, msg_type, msg_len) != -1;
+}
+
+/*
+ * Check if message is a CompressedData.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_compressed_msg(char msg_type)
+{
+ return msg_type == ZPQ_COMPRESSED_MSG_TYPE;
+}
+
+/*
+ * Check if message is a SetCompressionMethod.
+ * Return true if it is, otherwise false.
+ * */
+static inline bool
+zpq_is_set_compression_msg(char msg_type)
+{
+ return msg_type == ZPQ_SET_COMPRESSION_MSG_TYPE;
+}
+
+ZpqStream *
+zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size)
+{
+ ZpqStream *zpq;
+
+ /* zpqStream needs at least one compressor */
+ if (n_compressors == 0 || compressors == NULL)
+ {
+ return NULL;
+ }
+ zpq = (ZpqStream *) malloc(sizeof(ZpqStream));
+
+ zpq->compressors = compressors;
+ zpq->n_compressors = n_compressors;
+ zpq->compress_alg_idx = -1;
+ zpq->decompress_alg_idx = -1;
+
+ zpq->is_compressing = false;
+ zpq->is_decompressing = false;
+ zpq->rx_msg_bytes_left = 0;
+ zpq->tx_msg_bytes_left = 0;
+ zpq_buf_init(&zpq->tx_in);
+
+ zpq->tx_total = 0;
+ zpq->tx_total_raw = 0;
+ zpq->rx_total = 0;
+ zpq->rx_total_raw = 0;
+
+ zpq_buf_init(&zpq->rx_in);
+ zpq_buf_size_advance(&zpq->rx_in, rx_data_size);
+ Assert(rx_data_size < ZPQ_BUFFER_SIZE);
+ memcpy(zpq->rx_in.buf, rx_data, rx_data_size);
+
+ zpq_buf_init(&zpq->tx_out);
+
+ zpq->rx_func = rx_func;
+ zpq->tx_func = tx_func;
+ zpq->arg = arg;
+ zpq->reading_set_compression = false;
+ zpq->c_stream = NULL;
+ zpq->d_stream = NULL;
+
+ zpq_build_msg_compression_map(zpq);
+
+ return zpq;
+}
+
+/* Compress up to src_size bytes from *src into CompressedData and write it to the tx buffer.
+ * Returns ZS_OK on success, ZS_COMPRESS_ERROR if encountered a compression error. */
+static inline ssize_t
+zpq_write_compressed_message(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ size_t compressed_len;
+ ssize_t rc;
+ uint32 size;
+
+ /* check if have enough space */
+ if (zpq_buf_left(&zpq->tx_out) <= 5)
+ {
+ /* too little space for CompressedData, abort */
+ *src_processed = 0;
+ return ZS_OK;
+ }
+
+ compressed_len = 0;
+ rc = zs_write(zpq->c_stream, src, src_size, src_processed,
+ zpq_buf_size(&zpq->tx_out) + 5, zpq_buf_left(&zpq->tx_out) - 5, &compressed_len);
+
+ if (compressed_len > 0)
+ {
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_COMPRESSED_MSG_TYPE;
+ size = pg_hton32(compressed_len + 4);
+
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &size, sizeof(uint32)); /* write msg length */
+ compressed_len += 5; /* append header length to compressed data
+ * length */
+ }
+
+ zpq->tx_total_raw += *src_processed;
+ zpq->tx_total += compressed_len;
+ zpq_buf_size_advance(&zpq->tx_out, compressed_len);
+ return rc;
+}
+
+/* Copy the data directly from *src to the tx buffer */
+static void
+zpq_write_uncompressed(ZpqStream * zpq, char const *src, size_t src_size, size_t *src_processed)
+{
+ src_size = Min(zpq_buf_left(&zpq->tx_out), src_size);
+ memcpy(zpq_buf_size(&zpq->tx_out), src, src_size);
+
+ zpq->tx_total_raw += src_size;
+ zpq->tx_total += src_size;
+ zpq_buf_size_advance(&zpq->tx_out, src_size);
+ *src_processed = src_size;
+}
+
+static ssize_t
+zpq_write_set_compression_msg(ZpqStream * zpq, int new_compress_idx)
+{
+ uint32 len;
+ uint8 idx;
+
+ /*
+ * check if have enough space: msg_type(1 byte) + msg_len(4 bytes) +
+ * compress_alg_idx(1 byte)
+ */
+ if (zpq_buf_left(&zpq->tx_out) < 6)
+ {
+ return -1;
+ }
+
+ /* write CompressedData type */
+ *zpq_buf_size(&zpq->tx_out) = ZPQ_SET_COMPRESSION_MSG_TYPE;
+ len = pg_hton32(5);
+ memcpy(zpq_buf_size(&zpq->tx_out) + 1, &len, sizeof(uint32)); /* write msg length */
+
+ /* currently we expect idx to be in range [0, 255] */
+ Assert(new_compress_idx >= 0 && new_compress_idx <= UINT8_MAX);
+ idx = (uint8) new_compress_idx;
+ memcpy(zpq_buf_size(&zpq->tx_out) + 5, &idx, sizeof(uint8)); /* write
+ * new_compress_idx */
+
+ zpq->tx_total_raw += 6;
+ zpq->tx_total += 6;
+ zpq_buf_size_advance(&zpq->tx_out, 6);
+ return 0;
+}
+
+/* Determine if should compress the next message and change the current compression state */
+static ssize_t
+zpq_toggle_compression(ZpqStream * zpq, char msg_type, uint32 msg_len)
+{
+ int new_compress_idx = zpq_choose_compressor(zpq, msg_type, msg_len);
+ bool should_compress = new_compress_idx != -1;
+
+ /*
+ * negative new_compress_idx indicates that we should not compress this
+ * message
+ */
+ if (should_compress)
+ {
+ /*
+ * if the new compressor does not match the current one, process the
+ * switch
+ */
+ if (zpq->compress_alg_idx != new_compress_idx)
+ {
+ if (zpq_write_set_compression_msg(zpq, new_compress_idx))
+ {
+ /*
+ * come back later when we can write the entire
+ * SetCompressionMethod message
+ */
+ return 0;
+ }
+
+ zs_compressor_free(zpq->c_stream);
+ zpq->c_stream = zs_create_compressor(zpq->compressors[new_compress_idx].impl, zpq->compressors[new_compress_idx].level);
+ if (zpq->c_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->compress_alg_idx = new_compress_idx;
+ }
+ }
+
+ zpq->is_compressing = should_compress;
+ zpq->tx_msg_bytes_left = msg_len + 1;
+ return 0;
+}
+
+/*
+ * Internal write function. Reads the data from *src buffer,
+ * determines the postgres messages type and length.
+ * If message matches the compression criteria, it wraps the message into
+ * CompressedData. Otherwise, leaves the message unchanged.
+ * If *src data ends with incomplete message header, this function is not
+ * going to read this message header.
+ * Returns number of written raw bytes or error code.
+ * In the last case number of bytes written is stored in *processed.
+ */
+static ssize_t
+zpq_write_internal(ZpqStream * zpq, void const *src, size_t src_size, size_t *processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ do
+ {
+ /*
+ * try to read ahead the next message types and increase
+ * tx_msg_bytes_left, if possible
+ */
+ while (zpq->tx_msg_bytes_left > 0 && src_size - src_pos >= zpq->tx_msg_bytes_left + 5)
+ {
+ char msg_type = *((char *) src + src_pos + zpq->tx_msg_bytes_left);
+ uint32 msg_len;
+
+ memcpy(&msg_len, (char *) src + src_pos + zpq->tx_msg_bytes_left + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ if (zpq_should_compress(zpq, msg_type, msg_len) != zpq->is_compressing)
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+ zpq->tx_msg_bytes_left += msg_len + 1;
+ }
+
+ /*
+ * Write CompressedData if currently is compressing or have some
+ * buffered data left in underlying compression stream
+ */
+ if (zs_buffered(zpq->c_stream) || (zpq->is_compressing && zpq->tx_msg_bytes_left > 0))
+ {
+ size_t buf_processed = 0;
+ size_t to_compress = Min(zpq->tx_msg_bytes_left, src_size - src_pos);
+
+ rc = zpq_write_compressed_message(zpq, (char *) src + src_pos, to_compress, &buf_processed);
+ src_pos += buf_processed;
+ zpq->tx_msg_bytes_left -= buf_processed;
+
+ if (rc != ZS_OK)
+ {
+ *processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * If not going to compress the data from *src, just write it
+ * uncompressed.
+ */
+ else if (zpq->tx_msg_bytes_left > 0)
+ { /* determine next message type */
+ size_t copy_len = Min(src_size - src_pos, zpq->tx_msg_bytes_left);
+ size_t copy_processed = 0;
+
+ zpq_write_uncompressed(zpq, (char *) src + src_pos, copy_len, ©_processed);
+ src_pos += copy_processed;
+ zpq->tx_msg_bytes_left -= copy_processed;
+ }
+
+ /*
+ * Reached the compression toggle point, fetch next message header to
+ * determine compression state.
+ */
+ else
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ if (src_size - src_pos < 5)
+ {
+ /*
+ * must return here because we can't continue without full
+ * message header
+ */
+ *processed = src_pos;
+ return ZPQ_INCOMPLETE_HEADER;
+ }
+
+ msg_type = *((char *) src + src_pos);
+ memcpy(&msg_len, (char *) src + src_pos + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+ rc = zpq_toggle_compression(zpq, msg_type, msg_len);
+ if (rc)
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * repeat sending while there is some data in input or internal
+ * compression buffer
+ */
+ } while (src_pos < src_size && zpq_buf_left(&zpq->tx_out) > 6);
+
+ return src_pos;
+}
+
+ssize_t
+zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed)
+{
+ size_t src_pos = 0;
+ ssize_t rc;
+
+ /* try to process as much data as possible before calling the tx_func */
+ while (zpq_buf_left(&zpq->tx_out) > 6)
+ {
+ size_t copy_len = Min(zpq_buf_left(&zpq->tx_in), src_size - src_pos);
+ size_t processed;
+
+ memcpy(zpq_buf_size(&zpq->tx_in), (char *) src + src_pos, copy_len);
+ zpq_buf_size_advance(&zpq->tx_in, copy_len);
+ src_pos += copy_len;
+
+ if (zpq_buf_unread(&zpq->tx_in) == 0 && !zs_buffered(zpq->c_stream))
+ {
+ break;
+ }
+
+ processed = 0;
+
+ rc = zpq_write_internal(zpq, zpq_buf_pos(&zpq->tx_in), zpq_buf_unread(&zpq->tx_in), &processed);
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, rc);
+ zpq_buf_reuse(&zpq->tx_in);
+ }
+ else
+ {
+ zpq_buf_pos_advance(&zpq->tx_in, processed);
+ zpq_buf_reuse(&zpq->tx_in);
+ if (rc == ZPQ_INCOMPLETE_HEADER)
+ {
+ break;
+ }
+ *src_processed = src_pos;
+ return rc;
+ }
+ }
+
+ /*
+ * call the tx_func if have any bytes to send
+ */
+ while (zpq_buf_unread(&zpq->tx_out))
+ {
+ rc = zpq->tx_func(zpq->arg, zpq_buf_pos(&zpq->tx_out), zpq_buf_unread(&zpq->tx_out));
+ if (rc > 0)
+ {
+ zpq_buf_pos_advance(&zpq->tx_out, rc);
+ }
+ else
+ {
+ *src_processed = src_pos;
+ zpq_buf_reuse(&zpq->tx_out);
+ return rc;
+ }
+ }
+
+ zpq_buf_reuse(&zpq->tx_out);
+ return src_pos;
+}
+
+/* Decompress bytes from RX buffer and write up to dst_len of uncompressed data to *dst.
+ * Returns:
+ * ZS_OK on success,
+ * ZS_STREAM_END if reached end of compressed chunk
+ * ZS_DECOMPRESS_ERROR if encountered a decompression error */
+static inline ssize_t
+zpq_read_compressed_message(ZpqStream * zpq, char *dst, size_t dst_len, size_t *dst_processed)
+{
+ size_t rx_processed = 0;
+ ssize_t rc;
+ size_t read_len = Min(zpq->rx_msg_bytes_left, zpq_buf_unread(&zpq->rx_in));
+
+ Assert(read_len == zpq->rx_msg_bytes_left);
+ rc = zs_read(zpq->d_stream, zpq_buf_pos(&zpq->rx_in), read_len, &rx_processed,
+ dst, dst_len, dst_processed);
+
+ zpq_buf_pos_advance(&zpq->rx_in, rx_processed);
+ zpq->rx_total_raw += *dst_processed;
+ zpq->rx_msg_bytes_left -= rx_processed;
+ return rc;
+}
+
+/* Copy up to dst_len bytes from rx buffer to *dst.
+ * Returns amount of bytes copied. */
+static inline size_t
+zpq_read_uncompressed(ZpqStream * zpq, char *dst, size_t dst_len)
+{
+ size_t copy_len;
+
+ Assert(zpq_buf_unread(&zpq->rx_in) > 0);
+ copy_len = Min(zpq->rx_msg_bytes_left, Min(zpq_buf_unread(&zpq->rx_in), dst_len));
+
+ memcpy(dst, zpq_buf_pos(&zpq->rx_in), copy_len);
+
+ zpq_buf_pos_advance(&zpq->rx_in, copy_len);
+ zpq->rx_total_raw += copy_len;
+ zpq->rx_msg_bytes_left -= copy_len;
+ return copy_len;
+}
+
+/* Determine if should decompress the next message and
+ * change the current decompression state */
+static inline void
+zpq_toggle_decompression(ZpqStream * zpq)
+{
+ uint32 msg_len;
+ char msg_type = *zpq_buf_pos(&zpq->rx_in);
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + 1, 4);
+ msg_len = pg_ntoh32(msg_len);
+
+ if (zpq_is_set_compression_msg(msg_type))
+ {
+ Assert(msg_len == 5);
+ zpq->reading_set_compression = true;
+ /* set compression message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ }
+ else
+ {
+ zpq->is_decompressing = zpq_is_compressed_msg(msg_type);
+ zpq->rx_msg_bytes_left = msg_len + 1;
+
+ if (zpq->is_decompressing)
+ {
+ /* compressed message header is no longer needed, just skip it */
+ zpq_buf_pos_advance(&zpq->rx_in, 5);
+ zpq->rx_msg_bytes_left -= 5;
+ }
+ }
+}
+
+static inline ssize_t
+zpq_process_switch(ZpqStream * zpq)
+{
+ uint8 algorithm_idx;
+
+ if (zpq_buf_unread(&zpq->rx_in) < 1)
+ {
+ return 0;
+ }
+
+ algorithm_idx = *zpq_buf_pos(&zpq->rx_in);
+
+ zpq_buf_pos_advance(&zpq->rx_in, 1);
+ zpq->reading_set_compression = false;
+
+ if (algorithm_idx != zpq->decompress_alg_idx)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ zpq->d_stream = zs_create_decompressor(zpq->compressors[algorithm_idx].impl);
+ if (zpq->d_stream == NULL)
+ {
+ return ZPQ_FATAL_ERROR;
+ }
+ zpq->decompress_alg_idx = algorithm_idx;
+ }
+
+ return 0;
+}
+
+ssize_t
+zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock)
+{
+ size_t dst_pos = 0;
+ size_t dst_processed = 0;
+ ssize_t rc;
+
+ /* Read until some data fetched */
+ while (dst_pos == 0)
+ {
+ zpq_buf_reuse(&zpq->rx_in);
+
+ if (!zpq_buffered_rx(zpq) || (zpq->is_decompressing && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left))
+ {
+ if (noblock)
+ {
+ /*
+ * can't read anything w/o the potentially blocking backend
+ * call
+ */
+ return dst_pos;
+ }
+ rc = zpq->rx_func(zpq->arg, zpq_buf_size(&zpq->rx_in), zpq_buf_left(&zpq->rx_in));
+ if (rc > 0) /* read fetches some data */
+ {
+ zpq->rx_total += rc;
+ zpq_buf_size_advance(&zpq->rx_in, rc);
+ }
+ else /* read failed */
+ {
+ return rc;
+ }
+ }
+
+ /*
+ * try to read ahead the next message types and increase
+ * rx_msg_bytes_left, if possible (ONLY UNCOMPRESSED MESSAGES)
+ */
+ while (!zpq->is_decompressing && zpq->rx_msg_bytes_left > 0 && (zpq_buf_unread(&zpq->rx_in) >= zpq->rx_msg_bytes_left + 5))
+ {
+ char msg_type;
+ uint32 msg_len;
+
+ msg_type = *(zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left);
+ if (zpq_is_compressed_msg(msg_type) || zpq_is_set_compression_msg(msg_type))
+ {
+ /*
+ * cannot proceed further, encountered compression toggle
+ * point
+ */
+ break;
+ }
+
+ memcpy(&msg_len, zpq_buf_pos(&zpq->rx_in) + zpq->rx_msg_bytes_left + 1, 4);
+ zpq->rx_msg_bytes_left += pg_ntoh32(msg_len) + 1;
+ }
+
+
+ if (zpq->rx_msg_bytes_left > 0 || zs_buffered(zpq->d_stream))
+ {
+ dst_processed = 0;
+ if (zpq->is_decompressing || zs_buffered(zpq->d_stream))
+ {
+ if (!zs_buffered(zpq->d_stream) && zpq_buf_unread(&zpq->rx_in) < zpq->rx_msg_bytes_left)
+ {
+ /*
+ * prefer to read only the fully compressed messages or
+ * read if some data is buffered
+ */
+ continue;
+ }
+ rc = zpq_read_compressed_message(zpq, dst, dst_size - dst_pos, &dst_processed);
+ dst_pos += dst_processed;
+ if (rc == ZS_STREAM_END)
+ {
+ continue;
+ }
+ if (rc != ZS_OK)
+ {
+ return rc;
+ }
+ }
+ else
+ dst_pos += zpq_read_uncompressed(zpq, dst, dst_size - dst_pos);
+ }
+ else if (zpq->reading_set_compression)
+ {
+ zpq_process_switch(zpq);
+ }
+ else if (zpq_buf_unread(&zpq->rx_in) >= 5)
+ zpq_toggle_decompression(zpq);
+ }
+ return dst_pos;
+}
+
+bool
+zpq_buffered_rx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->rx_in) >= 5 || (zpq_buf_unread(&zpq->rx_in) > 0 && zpq->rx_msg_bytes_left > 0) ||
+ zs_buffered(zpq->d_stream) : 0;
+}
+
+bool
+zpq_buffered_tx(ZpqStream * zpq)
+{
+ return zpq ? zpq_buf_unread(&zpq->tx_in) >= 5 || (zpq_buf_unread(&zpq->tx_in) > 0 && zpq->tx_msg_bytes_left > 0) || zpq_buf_unread(&zpq->tx_out) > 0 ||
+ zs_buffered(zpq->c_stream) : 0;
+}
+
+void
+zpq_free(ZpqStream * zpq)
+{
+ if (zpq)
+ {
+ if (zpq->c_stream)
+ {
+ zs_compressor_free(zpq->c_stream);
+ }
+ if (zpq->d_stream)
+ {
+ zs_decompressor_free(zpq->d_stream);
+ }
+ free(zpq);
+ }
+}
+
+char const *
+zpq_compress_error(ZpqStream * zpq)
+{
+ return zs_compress_error(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_error(ZpqStream * zpq)
+{
+ return zs_decompress_error(zpq->d_stream);
+}
+
+char const *
+zpq_compress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_compress_algorithm_name(zpq->c_stream);
+}
+
+char const *
+zpq_decompress_algorithm_name(ZpqStream * zpq)
+{
+ return zs_decompress_algorithm_name(zpq->d_stream);
+}
+
+char *
+zpq_algorithms(ZpqStream * zpq)
+{
+ return zpq_serialize_compressors(zpq->compressors, zpq->n_compressors);
+}
+
+int
+zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int i;
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *protocol_extension = strchr(val, ';');
+
+ *compressors = NULL;
+ *n_compressors = 0;
+
+ /* No protocol extensions are currently supported */
+ if (protocol_extension)
+ *protocol_extension = '\0';
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ if (pg_strcasecmp(val, "true") == 0 ||
+ pg_strcasecmp(val, "yes") == 0 ||
+ pg_strcasecmp(val, "on") == 0 ||
+ pg_strcasecmp(val, "any") == 0 ||
+ pg_strcasecmp(val, "1") == 0)
+ {
+ /* return all available compressors */
+ *n_compressors = n_supported_algorithms;
+
+ if (n_supported_algorithms)
+ {
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+ for (i = 0; i < n_supported_algorithms; i++)
+ {
+ (*compressors)[i].impl = i;
+ (*compressors)[i].level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ }
+ }
+ return 1;
+ }
+
+ if (*val == 0 ||
+ pg_strcasecmp(val, "false") == 0 ||
+ pg_strcasecmp(val, "no") == 0 ||
+ pg_strcasecmp(val, "off") == 0 ||
+ pg_strcasecmp(val, "0") == 0)
+ {
+ /* Compression is disabled */
+ return 0;
+ }
+
+ return zpq_deserialize_compressors(val, compressors, n_compressors) ? 1 : -1;
+}
+
+bool
+zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors)
+{
+ int selected_alg_mask = 0; /* bitmask of already selected
+ * algorithms to avoid duplicates in
+ * compressors */
+ char **supported_algorithms = zs_get_supported_algorithms();
+ size_t n_supported_algorithms = 0;
+ char *c_string_dup = strdup(c_string); /* following parsing can
+ * modify the string */
+ char *p = c_string_dup;
+
+ *n_compressors = 0;
+
+ while (supported_algorithms[n_supported_algorithms] != NULL)
+ {
+ n_supported_algorithms += 1;
+ }
+
+ *compressors = malloc(n_supported_algorithms * sizeof(zpq_compressor));
+
+ while (*p != '\0')
+ {
+ char *sep = strchr(p, ',');
+ char *col;
+ int compression_level = ZPQ_DEFAULT_COMPRESSION_LEVEL;
+ bool found;
+
+ if (sep != NULL)
+ *sep = '\0';
+
+ col = strchr(p, ':');
+ if (col != NULL)
+ {
+ *col = '\0';
+ if (sscanf(col + 1, "%d", &compression_level) != 1)
+ {
+ pg_log_warning("invalid compression level %s in compression option '%s'", col + 1, p);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+ }
+ found = false;
+ for (int i = 0; supported_algorithms[i] != NULL; i++)
+ {
+ if (pg_strcasecmp(p, supported_algorithms[i]) == 0)
+ {
+ if (selected_alg_mask & (1 << i))
+ {
+ /* duplicates are not allowed */
+ pg_log_warning("duplicate algorithm %s in compressors string %s", p, c_string);
+ free(*compressors);
+ free(c_string_dup);
+ *compressors = NULL;
+ *n_compressors = 0;
+ return false;
+ }
+
+ (*compressors)[*n_compressors].impl = i;
+ (*compressors)[*n_compressors].level = compression_level;
+
+ selected_alg_mask |= 1 << i;
+ *n_compressors += 1;
+ found = true;
+ break;
+ }
+ }
+ if (!found)
+ {
+ pg_log_warning("algorithm %s is not supported", p);
+ }
+ if (sep)
+ p = sep + 1;
+ else
+ break;
+ }
+
+ if (*n_compressors == 0)
+ {
+ free(*compressors);
+ *compressors = NULL;
+ }
+ free(c_string_dup);
+ return true;
+}
+
+char *
+zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors)
+{
+ char *res;
+ char *p;
+ size_t i;
+ size_t total_len = 0;
+ char **supported_algorithms = zs_get_supported_algorithms();
+
+ if (n_compressors == 0)
+ {
+ return NULL;
+ }
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ size_t level_len;
+
+ if (!zs_is_valid_impl_id(compressors[i].impl))
+ {
+ pg_log_warning("algorithm impl_id %d is incorrect", compressors[i].impl);
+ return NULL;
+ }
+
+ /* determine the length of the compression level string */
+ level_len = compressors[i].level == 0 ? 1 : (int) floor(log10(abs(compressors[i].level))) + 1;
+ if (compressors[i].level < 0)
+ {
+ level_len += 1; /* add the leading "-" */
+ }
+
+ /*
+ * single entry looks like "alg_name:compression_level," so +2 is for
+ * ":" and "," symbols (or trailing null)
+ */
+ total_len += strlen(supported_algorithms[compressors[i].impl]) + level_len + 2;
+ }
+
+ res = p = malloc(total_len);
+
+ for (i = 0; i < n_compressors; i++)
+ {
+ p += sprintf(p, "%s:%d", supported_algorithms[compressors[i].impl], compressors[i].level);
+ if (i < n_compressors - 1)
+ *p++ = ',';
+ }
+ return res;
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a104caf4040..b4d3122c6eb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5347,9 +5347,9 @@
proname => 'pg_stat_get_activity', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => 'int4',
- proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8}',
- proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id}',
+ proallargtypes => '{int4,oid,int4,oid,text,text,text,text,text,timestamptz,timestamptz,timestamptz,timestamptz,inet,text,int4,xid,xid,text,bool,text,text,int4,text,numeric,text,bool,text,bool,int4,int8,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,datid,pid,usesysid,application_name,state,query,wait_event_type,wait_event,xact_start,query_start,backend_start,state_change,client_addr,client_hostname,client_port,backend_xid,backend_xmin,backend_type,ssl,sslversion,sslcipher,sslbits,ssl_client_dn,ssl_client_serial,ssl_issuer_dn,gss_auth,gss_princ,gss_enc,leader_pid,query_id,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
prosrc => 'pg_stat_get_activity' },
{ oid => '3318',
descr => 'statistics: information about progress of backends running maintenance command',
@@ -5668,6 +5668,14 @@
proargnames => '{stats_reset,prefetch,hit,skip_init,skip_new,skip_fpw,skip_rep,wal_distance,block_distance,io_depth}',
prosrc => 'pg_stat_get_recovery_prefetch' },
+{ oid => '9598', descr => 'statistics: information about network traffic',
+ proname => 'pg_stat_get_network_traffic', proisstrict => 'f', provolatile => 's',
+ proparallel => 'r', prorettype => 'record', proargtypes => 'int4',
+ proallargtypes => '{int4,int8,int8,int8,int8}',
+ proargmodes => '{i,o,o,o,o}',
+ proargnames => '{_beid,rx_raw_bytes,tx_raw_bytes,rx_compressed_bytes,tx_compressed_bytes}',
+ prosrc => 'pg_stat_get_network_traffic' },
+
{ oid => '2306', descr => 'statistics: information about SLRU caches',
proname => 'pg_stat_get_slru', prorows => '100', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
@@ -5809,6 +5817,10 @@
proname => 'pg_tablespace_location', provolatile => 's', prorettype => 'text',
proargtypes => 'oid', prosrc => 'pg_tablespace_location' },
+{ oid => '9257', descr => 'connection compression algorithm',
+ proname => 'pg_compression_algorithm', provolatile => 's', prorettype => 'text',
+ proargtypes => '', prosrc => 'pg_compression_algorithm' },
+
{ oid => '1946',
descr => 'convert bytea value into some ascii-only text string',
proname => 'encode', prorettype => 'text', proargtypes => 'bytea text',
diff --git a/src/include/common/z_stream.h b/src/include/common/z_stream.h
new file mode 100644
index 00000000000..4316563aa7f
--- /dev/null
+++ b/src/include/common/z_stream.h
@@ -0,0 +1,109 @@
+/*
+ * z_stream.h
+ * Streaming compression
+ */
+
+
+#ifndef Z_STREAM_H
+#define Z_STREAM_H
+
+#include <stdlib.h>
+
+#define ZS_OK (0)
+#define ZS_IO_ERROR (-1)
+#define ZS_DECOMPRESS_ERROR (-2)
+#define ZS_COMPRESS_ERROR (-3)
+#define ZS_STREAM_END (-4)
+#define ZS_DATA_PENDING (-5)
+#define ZS_INCOMPLETE_SRC (-6) /* cannot decompress unless full src message
+ * is fetched */
+#define ZS_EXPAND_DST (-7) /* cannot compress: dst buffer is too small
+ * for given src size */
+
+struct ZStream;
+typedef struct ZStream ZStream;
+
+#endif
+
+/*
+ * Create compression stream for sending compressed data.
+ * c_alg_impl: index of chosen compression algorithm
+ * c_level: compression c_level
+ */
+extern ZStream * zs_create_compressor(unsigned int c_alg_impl, int c_level);
+
+/*
+ * Create decompression stream for reading compressed data.
+ * d_alg_impl: index of chosen decompression algorithm
+ */
+extern ZStream * zs_create_decompressor(unsigned int d_alg_impl);
+
+/*
+ * Read up to "size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZS_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zs_read(ZStream * zs, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Write up to "size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZS_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *processed.
+ */
+extern ssize_t zs_write(ZStream * zs, void const *buf, size_t size, size_t *processed, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zs_decompress_error(ZStream * zs);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zs_compress_error(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal rx decompression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * Return true if non-flushed data might left in internal tx compression buffer.
+ */
+extern bool zs_buffered(ZStream * zs);
+
+/*
+ * End the compression stream.
+ */
+extern ssize_t zs_end_compression(ZStream * zs, void *dst, size_t dst_size, size_t *dst_processed);
+
+/*
+ * Free stream created by zs_create_compressor function.
+ */
+extern void zs_compressor_free(ZStream * zs);
+
+/*
+ * Free stream created by zs_create_decompressor function.
+ */
+extern void zs_decompressor_free(ZStream * zs);
+
+/*
+ * Get the name of chosen compression algorithm.
+ */
+extern char const *zs_compress_algorithm_name(ZStream * zs);
+
+/*
+ * Get the name of chosen decompression algorithm.
+ */
+extern char const *zs_decompress_algorithm_name(ZStream * zs);
+
+/*
+ Returns zero terminated array with compression algorithms names
+*/
+extern char **zs_get_supported_algorithms(void);
+
+/*
+ Returns true if provided id is a valid compression algorithm id, otherwise returns false
+*/
+extern bool zs_is_valid_impl_id(unsigned int id);
diff --git a/src/include/common/zpq_stream.h b/src/include/common/zpq_stream.h
new file mode 100644
index 00000000000..29b467015f2
--- /dev/null
+++ b/src/include/common/zpq_stream.h
@@ -0,0 +1,120 @@
+/*
+ * zpq_stream.h
+ * Streaming compression for libpq
+ */
+#include "z_stream.h"
+
+#ifndef ZPQ_STREAM_H
+#define ZPQ_STREAM_H
+
+#define ZPQ_DEFAULT_COMPRESSION_LEVEL (1)
+#define ZPQ_INCOMPLETE_HEADER (-6)
+#define ZPQ_FATAL_ERROR (-7)
+struct ZpqStream;
+typedef struct ZpqStream ZpqStream;
+
+typedef ssize_t (*zpq_tx_func) (void *arg, void const *data, size_t size);
+typedef ssize_t (*zpq_rx_func) (void *arg, void *data, size_t size);
+
+/*
+ * Descriptor of compression algorithm chosen by client
+ */
+typedef struct zpq_compressor
+{
+ unsigned int impl; /* compression algorithm index */
+ int level; /* compression level */
+} zpq_compressor;
+
+#endif
+
+/*
+ * Create compression stream with rx/tx function for reading/sending compressed data.
+ * tx_func: function for writing compressed data in underlying stream
+ * rx_func: function for reading compressed data from underlying stream
+ * arg: context passed to the function
+ * rx_data: received data (compressed data already fetched from input stream)
+ * rx_data_size: size of data fetched from input stream
+ */
+extern ZpqStream * zpq_create(zpq_compressor * compressors, size_t n_compressors, zpq_tx_func tx_func, zpq_rx_func rx_func, void *arg, char *rx_data, size_t rx_data_size);
+
+/*
+ * Write up to "src_size" raw (decompressed) bytes.
+ * Returns number of written raw bytes or error code.
+ * Error code is either ZPQ_COMPRESS_ERROR or error code returned by the tx function.
+ * In the last case number of bytes written is stored in *src_processed.
+ */
+extern ssize_t zpq_write(ZpqStream * zpq, void const *src, size_t src_size, size_t *src_processed);
+
+/*
+ * Read up to "dst_size" raw (decompressed) bytes.
+ * Returns number of decompressed bytes or error code.
+ * Error code is either ZPQ_DECOMPRESS_ERROR or error code returned by the rx function.
+ */
+extern ssize_t zpq_read(ZpqStream * zpq, void *dst, size_t dst_size, bool noblock);
+
+/*
+ * Return true if non-flushed data left in internal rx decompression buffer.
+ */
+extern bool zpq_buffered_rx(ZpqStream * zpq);
+
+/*
+ * Return true if non-flushed data left in internal tx compression buffer.
+ */
+extern bool zpq_buffered_tx(ZpqStream * zpq);
+
+/*
+ * Free stream created by zs_create function.
+ */
+extern void zpq_free(ZpqStream * zpq);
+
+/*
+ * Get decompressor error message.
+ */
+extern char const *zpq_decompress_error(ZpqStream * zpq);
+
+/*
+ * Get compressor error message.
+ */
+extern char const *zpq_compress_error(ZpqStream * zpq);
+
+/*
+ * Get the name of the current compression algorithm.
+ */
+extern char const *zpq_compress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Get the name of the current decompression algorithm.
+ */
+extern char const *zpq_decompress_algorithm_name(ZpqStream * zpq);
+
+/*
+ * Parse the compression setting. Returns:
+ * - 1 if the compression setting is valid
+ * - 0 if the compression setting is valid but disabled
+ * - -1 if the compression setting is invalid
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or if compression is disabled, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+extern int
+ zpq_parse_compression_setting(char *val, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Serialize the compressors array to string so it can be transmitted to the other side during the compression startup.
+ * For example, for array of two compressors (zstd, level 1), (zlib, level 2) resulting string would look like "zstd:1,zlib:2".
+ * Returns the resulting string.
+ */
+extern char
+ *zpq_serialize_compressors(zpq_compressor const *compressors, size_t n_compressors);
+
+/* Deserialize the compressors string received during the compression setup to a compressors array.
+ * For example, for string "zstd:1,zlib:2" compressors would be populated with 2 elements: (zstd, level 1), (zlib, level 2).
+ * Returns:
+ * - true if the compressors string is successfully parsed
+ * - false otherwise
+ * It also populates the compressors array with the recognized compressors. Size of the array is stored in n_compressors.
+ * If no supported compressors recognized or string is empty, then NULL is assigned to *compressors and n_compressors is set to 0.
+ */
+bool
+ zpq_deserialize_compressors(char const *c_string, zpq_compressor * *compressors, size_t *n_compressors);
+
+/* Return the currently enabled compression algorithms */
+char *zpq_algorithms(ZpqStream * zpq);
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 6d452ec6d95..b26808daa31 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -195,6 +195,9 @@ typedef struct Port
int keepalives_count;
int tcp_user_timeout;
+ char *compression_algorithms; /* Compression algorithms supported by
+ * client */
+
/*
* GSSAPI structures.
*/
diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h
index 2de7d9bad2c..0212a7d25f8 100644
--- a/src/include/libpq/libpq.h
+++ b/src/include/libpq/libpq.h
@@ -72,6 +72,7 @@ extern void StreamClose(pgsocket sock);
extern void TouchSocketFiles(void);
extern void RemoveSocketFiles(void);
extern void pq_init(void);
+extern int pq_configure(Port *port);
extern int pq_getbytes(char *s, size_t len);
extern void pq_startmsgread(void);
extern void pq_endmsgread(void);
diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h
index fcf68df39b9..98665015c51 100644
--- a/src/include/libpq/pqcomm.h
+++ b/src/include/libpq/pqcomm.h
@@ -99,6 +99,9 @@ typedef uint32 PacketLen;
extern PGDLLIMPORT bool Db_user_namespace;
+/* List of allowed compression algorithms */
+extern char *libpq_compress_algorithms;
+
/*
* In protocol 3.0 and later, the startup packet length is not fixed, but
* we set an arbitrary limit on it anyway. This is just to prevent simple
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index b582b46e9f9..1bda101f368 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -145,6 +145,12 @@ typedef struct PgBackendStatus
/* application name; MUST be null-terminated */
char *st_appname;
+ /* client-server traffic information */
+ uint64 st_rx_raw_bytes;
+ uint64 st_tx_raw_bytes;
+ uint64 st_rx_compressed_bytes;
+ uint64 st_tx_compressed_bytes;
+
/*
* Current command string; MUST be null-terminated. Note that this string
* possibly is truncated in the middle of a multi-byte character. As
@@ -309,6 +315,7 @@ extern void pgstat_report_query_id(uint64 query_id, bool force);
extern void pgstat_report_tempfile(size_t filesize);
extern void pgstat_report_appname(const char *appname);
extern void pgstat_report_xact_timestamp(TimestampTz tstamp);
+extern void pgstat_report_network_traffic(uint64 rx_raw_bytes, uint64 tx_raw_bytes, uint64 rx_compressed_bytes, uint64 tx_compressed_bytes);
extern const char *pgstat_get_backend_current_activity(int pid, bool checkUser);
extern const char *pgstat_get_crashed_backend_activity(int pid, char *buffer,
int buflen);
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 1d31b256fc9..27875cbee0f 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -30,6 +30,20 @@ endif
# The MSVC build system scrapes OBJS from this file. If you change any of
# the conditional additions of files to OBJS, update Mkvcbuild.pm to match.
+ifeq ($(with_lz4),yes)
+LIBS += -llz4
+SHLIB_LINK += -llz4
+endif
+
+ifeq ($(with_zlib),yes)
+LIBS += -lz
+SHLIB_LINK += -lz
+endif
+
+# We can't use Makefile variables here because the MSVC build system scrapes
+# OBJS from this file.
+
+
OBJS = \
$(WIN32RES) \
fe-auth-scram.o \
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index e8bcc883709..2759801b00e 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -186,3 +186,5 @@ PQpipelineStatus 183
PQsetTraceFlags 184
PQmblenBounded 185
PQsendFlushRequest 186
+PQcompression 187
+PQreadPending 188
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index a6120bf58b8..c50f9ac602c 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -25,6 +25,7 @@
#include "common/ip.h"
#include "common/link-canary.h"
#include "common/scram-common.h"
+#include "common/zpq_stream.h"
#include "common/string.h"
#include "fe-auth.h"
#include "libpq-fe.h"
@@ -336,6 +337,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
+ {"compression", "PGCOMPRESSION", "off", NULL,
+ "Libpq-compression", "", 16,
+ offsetof(struct pg_conn, compression)},
+
{"target_session_attrs", "PGTARGETSESSIONATTRS",
DefaultTargetSessionAttrs, NULL,
"Target-Session-Attrs", "", 15, /* sizeof("prefer-standby") = 15 */
@@ -445,6 +450,10 @@ pgthreadlock_t pg_g_threadlock = default_threadlock;
void
pqDropConnection(PGconn *conn, bool flushInput)
{
+ /* Release compression streams */
+ zpq_free(conn->zpqStream);
+ conn->zpqStream = NULL;
+
/* Drop any SSL state */
pqsecure_close(conn);
@@ -1371,6 +1380,36 @@ connectOptions2(PGconn *conn)
goto oom_error;
}
+ /*
+ * validate compression option
+ */
+ if (conn->compression && conn->compression[0])
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ int rc = zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors);
+
+ if (rc == -1)
+ {
+ conn->status = CONNECTION_BAD;
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("invalid %s value: \"%s\"\n"),
+ "compression", conn->compression);
+ return false;
+ }
+
+ if (rc == 1 && n_compressors == 0)
+ {
+ conn->status = CONNECTION_BAD;
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("no supported algorithms found, %s value: \"%s\"\n"),
+ "compression", conn->compression);
+ return false;
+ }
+
+ free(compressors);
+ }
+
/*
* validate target_session_attrs option, and set target_server_type
*/
@@ -3193,11 +3232,14 @@ keep_going: /* We will come back to here until there is
}
/*
- * Validate message type: we expect only an authentication
- * request or an error here. Anything else probably means
- * it's not Postgres on the other end at all.
+ * Validate message type. We expect only:
+ * - authentication request ('R')
+ * - error ('E')
+ * - protocol compression acknowledgment ('z')
+ * - NegotiateProtocolVersion in cases when server does not support protocol compression
+ * Anything else probably means it's not Postgres on the other end at all.
*/
- if (!(beresp == 'R' || beresp == 'E'))
+ if (!(beresp == 'R' || beresp == 'E' || beresp == 'z' || beresp == 'v'))
{
libpq_append_conn_error(conn, "expected authentication request from server, but received %c",
beresp);
@@ -3273,6 +3315,72 @@ keep_going: /* We will come back to here until there is
return PGRES_POLLING_READING;
}
+ if (beresp == 'z') /* Switch on compression */
+ {
+ zpq_compressor *compressors;
+ size_t n_compressors;
+ char *resp = malloc(msgLength);
+
+ pqGetnchar(resp, msgLength, conn);
+
+ if (!zpq_deserialize_compressors(resp, &compressors, &n_compressors))
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server returned unrecognized compression setting: %s\n"),
+ resp);
+ free(resp);
+ goto error_return;
+ }
+ free(resp);
+
+ if (n_compressors == 0)
+ {
+ /*
+ * If there are no compressors returned, it means that
+ * the server rejected all the proposed compression
+ * algorithms. Report an error and exit.
+ */
+ // conn->inStart = conn->inCursor;
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server rejected protocol compression\n"));
+ goto error_return;
+ // goto keep_going;
+ }
+
+ Assert(!conn->zpqStream);
+ conn->zpqStream = zpq_create(compressors, n_compressors,
+ (zpq_tx_func) pqsecure_write, (zpq_rx_func) pqsecure_read,
+ conn,
+ &conn->inBuffer[conn->inCursor],
+ conn->inEnd - conn->inCursor);
+ if (!conn->zpqStream)
+ {
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("failed to initialize compression\n"));
+ free(compressors);
+ goto error_return;
+ }
+ /* reset buffer */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+ else if (conn->n_compressors > 0 && !conn->zpqStream)
+ {
+ /*
+ * Despite the client requesting the compression, the
+ * backend did not reply with the CompressionACK message.
+ * This case covers either no reply at all or
+ * NegotiateProtocolVersion reply. If the backend supports
+ * the protocol compression feature, it should reply with
+ * CompressionACK in any case, even in case of compression
+ * request rejection. If the backend did not reply with
+ * the CompressionACK message, it means that it does not
+ * support this feature. Report an error and exit.
+ */
+ appendPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("server does not support protocol compression\n"));
+ goto error_return;
+ }
+
/* Handle errors. */
if (beresp == 'E')
{
@@ -3972,8 +4080,8 @@ freePGconn(PGconn *conn)
free(conn->connhost[i].password);
}
}
- free(conn->connhost);
+ free(conn->connhost);
free(conn->client_encoding_initial);
free(conn->events);
free(conn->pghost);
@@ -3986,7 +4094,9 @@ freePGconn(PGconn *conn)
free(conn->fbappname);
free(conn->dbName);
free(conn->replication);
+ free(conn->compression);
free(conn->pguser);
+
if (conn->pgpass)
{
explicit_bzero(conn->pgpass, strlen(conn->pgpass));
@@ -6590,6 +6700,15 @@ PQuser(const PGconn *conn)
return conn->pguser;
}
+char *
+PQcompression(const PGconn *conn)
+{
+ if (!conn || !conn->zpqStream)
+ return NULL;
+
+ return zpq_algorithms(conn->zpqStream);
+}
+
char *
PQpass(const PGconn *conn)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index da229d632a1..575e634639a 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2063,9 +2063,7 @@ PQgetResult(PGconn *conn)
* EOF indication. We expect therefore that this won't result in any
* undue delay in reporting a previous write failure.)
*/
- if (flushResult ||
- pqWait(true, false, conn) ||
- pqReadData(conn) < 0)
+ if (flushResult || pqWait(true, false, conn) || pqReadData(conn) < 0)
{
/* Report the error saved by pqWait or pqReadData */
pqSaveErrorResult(conn);
@@ -3854,6 +3852,12 @@ pqPipelineFlush(PGconn *conn)
return 0;
}
+int
+PQreadPending(PGconn *conn)
+{
+ return pqReadPending(conn);
+}
+
/*
* PQfreemem - safely frees memory allocated
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 4159610f6c7..d3a4e34d93e 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -51,12 +51,24 @@
#include "pg_config_paths.h"
#include "port/pg_bswap.h"
+#include <common/zpq_stream.h>
+
static int pqPutMsgBytes(const void *buf, size_t len, PGconn *conn);
static int pqSendSome(PGconn *conn, int len);
static int pqSocketCheck(PGconn *conn, int forRead, int forWrite,
time_t end_time);
static int pqSocketPoll(int sock, int forRead, int forWrite, time_t end_time);
+/*
+ * Use zpq_read if compression is switched on
+ */
+#define pq_read_conn(conn) \
+ (conn->zpqStream \
+ ? zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd, false) \
+ : pqsecure_read(conn, conn->inBuffer + conn->inEnd, \
+ conn->inBufSize - conn->inEnd))
+
/*
* PQlibVersion: return the libpq version number
*/
@@ -614,10 +626,17 @@ pqReadData(PGconn *conn)
/* OK, try to read some data */
retry3:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -709,10 +728,18 @@ retry3:
* arrived.
*/
retry4:
- nread = pqsecure_read(conn, conn->inBuffer + conn->inEnd,
- conn->inBufSize - conn->inEnd);
+ nread = pq_read_conn(conn);
+
if (nread < 0)
{
+ if (nread == ZS_DECOMPRESS_ERROR)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("decompress error: %s\n"),
+ zpq_decompress_error(conn->zpqStream));
+ return -1;
+ }
+
switch (SOCK_ERRNO)
{
case EINTR:
@@ -822,12 +849,18 @@ pqSendSome(PGconn *conn, int len)
}
/* while there's still data to send */
- while (len > 0)
+ while (len > 0 || zpq_buffered_tx(conn->zpqStream))
{
int sent;
+ size_t processed = 0;
+ /*
+ * Use zpq_write if compression is switched on
+ */
+ sent = conn->zpqStream
+ ? zpq_write(conn->zpqStream, ptr, len, &processed)
#ifndef WIN32
- sent = pqsecure_write(conn, ptr, len);
+ : pqsecure_write(conn, ptr, len);
#else
/*
@@ -835,8 +868,11 @@ pqSendSome(PGconn *conn, int len)
* failure-point appears to be different in different versions of
* Windows, but 64k should always be safe.
*/
- sent = pqsecure_write(conn, ptr, Min(len, 65536));
+: pqsecure_write(conn, ptr, Min(len, 65536));
#endif
+ ptr += processed;
+ len -= processed;
+ remaining -= processed;
if (sent < 0)
{
@@ -885,7 +921,7 @@ pqSendSome(PGconn *conn, int len)
remaining -= sent;
}
- if (len > 0)
+ if (len > 0 || sent < 0 || zpq_buffered_tx(conn->zpqStream))
{
/*
* We didn't send it all, wait till we can send more.
@@ -953,7 +989,7 @@ pqSendSome(PGconn *conn, int len)
int
pqFlush(PGconn *conn)
{
- if (conn->outCount > 0)
+ if (conn->outCount > 0 || zpq_buffered_tx(conn->zpqStream))
{
if (conn->Pfdebug)
fflush(conn->Pfdebug);
@@ -978,6 +1014,8 @@ pqFlush(PGconn *conn)
int
pqWait(int forRead, int forWrite, PGconn *conn)
{
+ if (forRead && conn->inCursor < conn->inEnd)
+ return 0;
return pqWaitTimed(forRead, forWrite, conn, (time_t) -1);
}
@@ -1034,6 +1072,9 @@ pqWriteReady(PGconn *conn)
*
* If SSL is in use, the SSL buffer is checked prior to checking the socket
* for read data directly.
+ *
+ * If ZPQ stream is in use, the ZPQ buffer is checked prior to checking
+ * the socket for read data directly.
*/
static int
pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
@@ -1048,14 +1089,10 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return -1;
}
-#ifdef USE_SSL
- /* Check for SSL library buffering read bytes */
- if (forRead && conn->ssl_in_use && pgtls_read_pending(conn))
+ if (forRead && (pqReadPending(conn) > 0))
{
- /* short-circuit the select */
return 1;
}
-#endif
/* We will retry as long as we get EINTR */
do
@@ -1073,6 +1110,33 @@ pqSocketCheck(PGconn *conn, int forRead, int forWrite, time_t end_time)
return result;
}
+/*
+ * Check if there is some data pending in ZPQ / SSL read buffers.
+ * Returns -1 on failure, 0 if no, 1 if yes.
+ */
+int
+pqReadPending(PGconn *conn)
+{
+ if (!conn)
+ return -1;
+
+ /* check for ZPQ stream buffered read bytes */
+ if (zpq_buffered_rx(conn->zpqStream))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+
+#ifdef USE_SSL
+ /* Check for SSL library buffering read bytes */
+ if (conn->ssl_in_use && pgtls_read_pending(conn))
+ {
+ /* short-circuit the select */
+ return 1;
+ }
+#endif
+ return 0;
+}
/*
* Check a file descriptor for read and/or write data, possibly waiting.
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 88dd360c905..e76a1a9b36c 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -69,6 +69,19 @@ pqParseInput3(PGconn *conn)
*/
for (;;)
{
+ /*
+ * Read the buffered compressed data w/o blocking
+ */
+ if (conn->zpqStream && pqReadPending(conn) && (conn->inBufSize - conn->inEnd > 0))
+ {
+ int rc = zpq_read(conn->zpqStream, conn->inBuffer + conn->inEnd, conn->inBufSize - conn->inEnd, true);
+
+ if (rc > 0)
+ {
+ conn->inEnd += rc;
+ }
+ }
+
/*
* Try to read a message. First get the type code and length. Return
* if not enough data.
@@ -1692,7 +1705,7 @@ pqGetCopyData3(PGconn *conn, char **buffer, int async)
if (msgLength == 0)
{
/* Don't block if async read requested */
- if (async)
+ if (async && !pqReadPending(conn))
return 0;
/* Need to load more data */
if (pqWait(true, false, conn) ||
@@ -2174,6 +2187,50 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
return startpacket;
}
+/*
+ * Build comma-separated list of compression algorithms requested by client.
+ * It can be either explicitly specified by user in connection string, or
+ * include all algorithms supported by client library.
+ * This function returns true if the compression string is successfully parsed and
+ * stores a comma-separated list of algorithms in *client_compressors.
+ * If compression is disabled, then NULL is assigned to *client_compressors.
+ * Also it creates an array of compressor descriptors, each element of which corresponds to
+ * the corresponding algorithm name in *client_compressors list. This array is stored in PGconn
+ * and is used during handshake when a compression acknowledgment response is received from the server.
+ */
+static bool
+build_compressors_list(PGconn *conn, char **client_compressors, bool build_descriptors)
+{
+ zpq_compressor *compressors;
+ size_t n_compressors;
+
+ if (!zpq_parse_compression_setting(conn->compression, &compressors, &n_compressors))
+ {
+ return false;
+ }
+
+ *client_compressors = NULL;
+ if (build_descriptors)
+ {
+ conn->compressors = compressors;
+ conn->n_compressors = n_compressors;
+ }
+
+ if (n_compressors == 0)
+ {
+ /* no compressors available, return */
+ return true;
+ }
+
+ *client_compressors = zpq_serialize_compressors(compressors, n_compressors);
+
+ if (!build_descriptors)
+ {
+ free(compressors);
+ }
+ return true;
+}
+
/*
* Build a startup packet given a filled-in PGconn structure.
*
@@ -2220,6 +2277,18 @@ build_startup_packet(const PGconn *conn, char *packet,
ADD_STARTUP_OPTION("replication", conn->replication);
if (conn->pgoptions && conn->pgoptions[0])
ADD_STARTUP_OPTION("options", conn->pgoptions);
+ if (conn->compression && conn->compression[0])
+ {
+ char *client_compression_algorithms;
+
+ if (build_compressors_list((PGconn *) conn, &client_compression_algorithms, packet == NULL))
+ {
+ if (client_compression_algorithms)
+ {
+ ADD_STARTUP_OPTION("_pq_.compression", client_compression_algorithms);
+ }
+ }
+ }
if (conn->send_appname)
{
/* Use appname if present, otherwise use fallback */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index b7df3224c0f..6fd24d2dda3 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -342,6 +342,7 @@ extern char *PQhostaddr(const PGconn *conn);
extern char *PQport(const PGconn *conn);
extern char *PQtty(const PGconn *conn);
extern char *PQoptions(const PGconn *conn);
+extern char *PQcompression(const PGconn *conn);
extern ConnStatusType PQstatus(const PGconn *conn);
extern PGTransactionStatusType PQtransactionStatus(const PGconn *conn);
extern const char *PQparameterStatus(const PGconn *conn,
@@ -500,6 +501,9 @@ extern PGPing PQpingParams(const char *const *keywords,
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
+extern int
+ PQreadPending(PGconn *conn);
+
/*
* "Fast path" interface --- not really recommended for application
* use
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index c24645b4696..43ba1c6ad88 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -41,6 +41,7 @@
/* include stuff common to fe and be */
#include "libpq/pqcomm.h"
+#include "common/zpq_stream.h"
/* include stuff found in fe only */
#include "fe-auth-sasl.h"
#include "pqexpbuffer.h"
@@ -343,6 +344,7 @@ typedef struct pg_conn_host
* found in password file. */
} pg_conn_host;
+
/*
* PGconn stores all the state data associated with a single connection
* to a backend.
@@ -395,6 +397,14 @@ struct pg_conn
* "sspi") */
char *ssl_min_protocol_version; /* minimum TLS protocol version */
char *ssl_max_protocol_version; /* maximum TLS protocol version */
+
+ char *compression; /* stream compression (boolean value, "any" or
+ * list of compression algorithms separated by
+ * comma) */
+ zpq_compressor *compressors; /* descriptors of compression algorithms
+ * chosen by client */
+ unsigned n_compressors; /* size of compressors array */
+
char *target_session_attrs; /* desired session properties */
/* Optional file to write trace info to */
@@ -590,6 +600,9 @@ struct pg_conn
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Compression stream */
+ ZpqStream *zpqStream;
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
@@ -722,6 +735,7 @@ extern int pqWaitTimed(int forRead, int forWrite, PGconn *conn,
time_t finish_time);
extern int pqReadReady(PGconn *conn);
extern int pqWriteReady(PGconn *conn);
+extern int pqReadPending(PGconn *conn);
/* === in fe-secure.c === */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 624d0e5aae1..606b72e9c1e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1756,7 +1756,7 @@ pg_stat_activity| SELECT s.datid,
s.query_id,
s.query,
s.backend_type
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
LEFT JOIN pg_database d ON ((s.datid = d.oid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_all_indexes| SELECT c.oid AS relid,
@@ -1871,8 +1871,14 @@ pg_stat_gssapi| SELECT s.pid,
s.gss_auth AS gss_authenticated,
s.gss_princ AS principal,
s.gss_enc AS encrypted
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
+pg_stat_network_traffic| SELECT s.pid,
+ s.rx_raw_bytes,
+ s.tx_raw_bytes,
+ s.rx_compressed_bytes,
+ s.tx_compressed_bytes
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes);
pg_stat_progress_analyze| SELECT s.pid,
s.datid,
d.datname,
@@ -2052,7 +2058,7 @@ pg_stat_replication| SELECT s.pid,
w.sync_priority,
w.sync_state,
w.reply_time
- FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_replication_slots| SELECT s.slot_name,
@@ -2086,7 +2092,7 @@ pg_stat_ssl| SELECT s.pid,
s.ssl_client_dn AS client_dn,
s.ssl_client_serial AS client_serial,
s.ssl_issuer_dn AS issuer_dn
- FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id)
+ FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid, query_id, rx_raw_bytes, tx_raw_bytes, rx_compressed_bytes, tx_compressed_bytes)
WHERE (s.client_port IS NOT NULL);
pg_stat_subscription| SELECT su.oid AS subid,
su.subname,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e404254..9dec3431ce9 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -139,7 +139,7 @@ sub mkvcbuild
keywords.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c
rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c
- username.c wait_error.c wchar.c);
+ username.c wait_error.c wchar.c z_stream.c zpq_stream.c);
if ($solution->{options}->{openssl})
{
--
2.25.1
0002-Add-ZSTD-support.patch (text/x-diff, 7.6K) download
From e32a9655ea7892c3f89abbdd4a401bc8172ef507 Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@protonmail.ch>
Date: Thu, 30 Dec 2021 15:59:01 +0500
Subject: [PATCH 2/4] Add ZSTD support
---
src/Makefile.global.in | 1 +
src/backend/Makefile | 4 +
src/common/z_stream.c | 228 ++++++++++++++++++++++++++++++++++
src/interfaces/libpq/Makefile | 5 +
src/tools/msvc/Solution.pm | 3 +-
5 files changed, 240 insertions(+), 1 deletion(-)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 1b3dc97e97d..da3abeddc94 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -197,6 +197,7 @@ with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
with_zlib = @with_zlib@
with_lz4 = @with_lz4@
+with_zstd = @with_zstd@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
enable_debug = @enable_debug@
diff --git a/src/backend/Makefile b/src/backend/Makefile
index f90e078b6ed..d7041c9aede 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -55,6 +55,10 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+endif
+
ifeq ($(with_lz4),yes)
LIBS += -llz4
endif
diff --git a/src/common/z_stream.c b/src/common/z_stream.c
index b75f194a1cf..98f4e4d1d9e 100644
--- a/src/common/z_stream.c
+++ b/src/common/z_stream.c
@@ -89,6 +89,231 @@ struct ZStream
bool not_flushed;
};
+#if HAVE_LIBZSTD
+
+#include <stdlib.h>
+#include <zstd.h>
+
+/*
+ * Maximum allowed back-reference distance, expressed as power of 2.
+ * This setting controls max compressor/decompressor window size.
+ * More details https://github.com/facebook/zstd/blob/v1.4.7/lib/zstd.h#L536
+ */
+#define ZSTD_WINDOWLOG_LIMIT 23 /* set max window size to 8MB */
+
+
+typedef struct ZS_ZSTD_CStream
+{
+ ZSTD_CStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_CStream;
+
+typedef struct ZS_ZSTD_DStream
+{
+ ZSTD_DStream *stream;
+ char const *error; /* error message */
+} ZS_ZSTD_DStream;
+
+static void *
+zstd_create_compressor(int level)
+{
+ size_t rc;
+ ZS_ZSTD_CStream *c_stream = (ZS_ZSTD_CStream *) malloc(sizeof(ZS_ZSTD_CStream));
+
+ c_stream->stream = ZSTD_createCStream();
+ rc = ZSTD_initCStream(c_stream->stream, level);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeCStream(c_stream->stream);
+ free(c_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_CCtx_setParameter(c_stream->stream, ZSTD_c_windowLog, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ c_stream->error = NULL;
+ return c_stream;
+}
+
+static void *
+zstd_create_decompressor()
+{
+ size_t rc;
+ ZS_ZSTD_DStream *d_stream = (ZS_ZSTD_DStream *) malloc(sizeof(ZS_ZSTD_DStream));
+
+ d_stream->stream = ZSTD_createDStream();
+ rc = ZSTD_initDStream(d_stream->stream);
+ if (ZSTD_isError(rc))
+ {
+ ZSTD_freeDStream(d_stream->stream);
+ free(d_stream);
+ return NULL;
+ }
+#if ZSTD_VERSION_MAJOR > 1 || ZSTD_VERSION_MINOR > 3
+ ZSTD_DCtx_setParameter(d_stream->stream, ZSTD_d_windowLogMax, ZSTD_WINDOWLOG_LIMIT);
+#endif
+ d_stream->error = NULL;
+ return d_stream;
+}
+
+static ssize_t
+zstd_decompress(void *d_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+ size_t rc;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ rc = ZSTD_decompressStream(ds->stream, &out, &in);
+
+ *src_processed = in.pos;
+ *dst_processed = out.pos;
+ if (ZSTD_isError(rc))
+ {
+ ds->error = ZSTD_getErrorName(rc);
+ return ZS_DECOMPRESS_ERROR;
+ }
+
+ if (rc == 0)
+ {
+ return ZS_STREAM_END;
+ }
+
+ if (out.pos == out.size)
+ {
+ /*
+ * if `output.pos == output.size`, there might be some data left
+ * within internal buffers
+ */
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_compress(void *c_stream, void const *src, size_t src_size, size_t *src_processed, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_inBuffer in;
+ ZSTD_outBuffer out;
+
+ in.src = src;
+ in.pos = 0;
+ in.size = src_size;
+
+ out.dst = dst;
+ out.pos = 0;
+ out.size = dst_size;
+
+ if (in.pos < src_size) /* Has something to compress in input buffer */
+ {
+ size_t rc = ZSTD_compressStream(cs->stream, &out, &in);
+
+ *dst_processed = out.pos;
+ *src_processed = in.pos;
+ if (ZSTD_isError(rc))
+ {
+ cs->error = ZSTD_getErrorName(rc);
+ return ZS_COMPRESS_ERROR;
+ }
+ }
+
+ if (in.pos == src_size) /* All data is compressed: flush internal zstd
+ * buffer */
+ {
+ size_t tx_not_flushed = ZSTD_flushStream(cs->stream, &out);
+
+ *dst_processed = out.pos;
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ }
+
+ return ZS_OK;
+}
+
+static ssize_t
+zstd_end(void *c_stream, void *dst, size_t dst_size, size_t *dst_processed)
+{
+ size_t tx_not_flushed;
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+ ZSTD_outBuffer output;
+
+ output.dst = dst;
+ output.pos = 0;
+ output.size = dst_size;
+
+ do
+ {
+ tx_not_flushed = ZSTD_endStream(cs->stream, &output);
+ } while ((tx_not_flushed > 0) && (output.pos < output.size));
+
+ *dst_processed = output.pos;
+
+ if (tx_not_flushed > 0)
+ {
+ return ZS_DATA_PENDING;
+ }
+ return ZS_OK;
+}
+
+static void
+zstd_free_compressor(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ if (cs != NULL)
+ {
+ ZSTD_freeCStream(cs->stream);
+ free(cs);
+ }
+}
+
+static void
+zstd_free_decompressor(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ if (ds != NULL)
+ {
+ ZSTD_freeDStream(ds->stream);
+ free(ds);
+ }
+}
+
+static char const *
+zstd_compress_error(void *c_stream)
+{
+ ZS_ZSTD_CStream *cs = (ZS_ZSTD_CStream *) c_stream;
+
+ return cs->error;
+}
+
+static char const *
+zstd_decompress_error(void *d_stream)
+{
+ ZS_ZSTD_DStream *ds = (ZS_ZSTD_DStream *) d_stream;
+
+ return ds->error;
+}
+
+static char const *
+zstd_name(void)
+{
+ return "zstd";
+}
+
+#endif
+
#if HAVE_LIBZ
#include <stdlib.h>
@@ -424,6 +649,9 @@ no_compression_name(void)
*/
static ZAlgorithm const zs_algorithms[] =
{
+#if HAVE_LIBZSTD
+ {zstd_name, zstd_create_compressor, zstd_create_decompressor, zstd_decompress, zstd_compress, zstd_free_compressor, zstd_free_decompressor, zstd_compress_error, zstd_decompress_error, zstd_end},
+#endif
#if HAVE_LIBZ
{zlib_name, zlib_create_compressor, zlib_create_decompressor, zlib_decompress, zlib_compress, zlib_free_compressor, zlib_free_decompressor, zlib_error, zlib_error, zlib_end},
#endif
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 27875cbee0f..9b1db2c0dc5 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -35,6 +35,11 @@ LIBS += -llz4
SHLIB_LINK += -llz4
endif
+ifeq ($(with_zstd),yes)
+LIBS += -lzstd
+SHLIB_LINK += -lzstd
+endif
+
ifeq ($(with_zlib),yes)
LIBS += -lz
SHLIB_LINK += -lz
diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm
index c2acb58df0e..f35b29f8ebd 100644
--- a/src/tools/msvc/Solution.pm
+++ b/src/tools/msvc/Solution.pm
@@ -293,7 +293,8 @@ sub GenerateFiles
HAVE_LIBXML2 => undef,
HAVE_LIBXSLT => undef,
HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef,
- HAVE_LIBZSTD => undef,
+ HAVE_LIBZSTD => $self->{options}->{zstd} ? 1 : undef,
+ HAVE_LINK => undef,
HAVE_LOCALE_T => 1,
HAVE_LONG_INT_64 => undef,
HAVE_LONG_LONG_INT_64 => 1,
--
2.25.1
0003-Turn-on-zlib-compression-for-CI-test-runs.patch (text/x-diff, 2.3K) download
From 707f406252e2b38699677081bd0da9bf5a19b04e Mon Sep 17 00:00:00 2001
From: usernamedt <usernamedt@yandex-team.com>
Date: Wed, 12 Jan 2022 23:14:02 +0500
Subject: [PATCH 3/4] Turn on zlib compression for CI test runs
---
src/backend/utils/misc/guc_tables.c | 2 +-
src/common/zpq_stream.c | 11 +++++------
src/interfaces/libpq/fe-connect.c | 2 +-
3 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index bef045e5be9..e1e5cab63af 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -4453,7 +4453,7 @@ struct config_string ConfigureNamesString[] =
NULL
},
&libpq_compress_algorithms,
- "off",
+ "zlib",
check_libpq_compression, NULL, NULL
},
diff --git a/src/common/zpq_stream.c b/src/common/zpq_stream.c
index a712200e453..9bdb3527072 100644
--- a/src/common/zpq_stream.c
+++ b/src/common/zpq_stream.c
@@ -156,9 +156,11 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
for (i = 0; i < 256; i++)
{
- zpq->compressor_by_msg_type[i] = -1;
+ /* Compress all messages during the CI test runs */
+ zpq->compressor_by_msg_type[i] = 0;
}
+ // XXX why is this looping around compressors and overwriting the same bytes??
for (i = 0; i < zpq->n_compressors; i++)
{
/* compress CopyData, DataRow and Query messages */
@@ -186,11 +188,8 @@ zpq_build_msg_compression_map(ZpqStream * zpq)
static inline int
zpq_choose_compressor(ZpqStream * zpq, char msg_type, uint32 msg_len)
{
- if (msg_len >= ZPQ_COMPRESS_THRESHOLD)
- {
- return zpq->compressor_by_msg_type[(unsigned char) msg_type];
- }
- return -1;
+ /* Compress messages with any length during the CI test runs */
+ return zpq->compressor_by_msg_type[(unsigned char) msg_type];
}
/*
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index c50f9ac602c..f8f831f6665 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -337,7 +337,7 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"Replication", "D", 5,
offsetof(struct pg_conn, replication)},
- {"compression", "PGCOMPRESSION", "off", NULL,
+ {"compression", "PGCOMPRESSION", "zlib", NULL,
"Libpq-compression", "", 16,
offsetof(struct pg_conn, compression)},
--
2.25.1
0004-rebase-and-convert-to-meson.patch (text/x-diff, 4.0K) download
From 7f49573af55ea8824c919cf12271175cb8983497 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <pryzbyj@telsasoft.com>
Date: Sat, 12 Nov 2022 16:20:59 -0600
Subject: [PATCH 4/4] rebase and convert to meson
---
src/backend/Makefile | 12 ------------
src/backend/libpq/pqcomm.c | 4 ++--
src/backend/utils/misc/guc_funcs.c | 4 ++--
src/backend/utils/misc/guc_tables.c | 4 +++-
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/common/meson.build | 4 +++-
6 files changed, 11 insertions(+), 18 deletions(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index d7041c9aede..181c217fae4 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -55,18 +55,6 @@ ifeq ($(with_systemd),yes)
LIBS += -lsystemd
endif
-ifeq ($(with_zstd),yes)
-LIBS += -lzstd
-endif
-
-ifeq ($(with_lz4),yes)
-LIBS += -llz4
-endif
-
-ifeq ($(with_zlib),yes)
-LIBS += -lz
-endif
-
##########################################################################
all: submake-libpgport submake-catalog-headers submake-utils-headers postgres $(POSTGRES_IMP)
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 9bf4672950c..55fede3dc7d 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -111,7 +111,7 @@ int Unix_socket_permissions;
char *Unix_socket_group;
/* GUC variable containing the allowed compression algorithms list (separated by comma) */
-char *libpq_compress_algorithms;
+char *libpq_compress_algorithms = "on";
/* Where the Unix socket files are (list of palloc'd strings) */
static List *sock_paths = NIL;
@@ -2209,7 +2209,7 @@ retry:
return true;
}
-PG_FUNCTION_INFO_V1(pg_compression_algorithm);
+// PG_FUNCTION_INFO_V1(pg_compression_algorithm);
Datum
pg_compression_algorithm(PG_FUNCTION_ARGS)
diff --git a/src/backend/utils/misc/guc_funcs.c b/src/backend/utils/misc/guc_funcs.c
index 5585f992702..767764bfcbf 100644
--- a/src/backend/utils/misc/guc_funcs.c
+++ b/src/backend/utils/misc/guc_funcs.c
@@ -1052,9 +1052,9 @@ show_all_file_settings(PG_FUNCTION_ARGS)
#include "common/zpq_stream.h"
-static bool check_libpq_compression(char **newval, void **extra, GucSource source);
+bool check_libpq_compression(char **newval, void **extra, GucSource source);
-static bool
+bool
check_libpq_compression(char **newval, void **extra, GucSource source)
{
zpq_compressor *compressors;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index e1e5cab63af..d270d339e0e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3691,6 +3691,8 @@ struct config_real ConfigureNamesReal[] =
};
+extern bool check_libpq_compression(char **newval, void **extra, GucSource source);
+
struct config_string ConfigureNamesString[] =
{
{
@@ -4453,7 +4455,7 @@ struct config_string ConfigureNamesString[] =
NULL
},
&libpq_compress_algorithms,
- "zlib",
+ "on",
check_libpq_compression, NULL, NULL
},
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 868d21c351e..da3d0bfcc16 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -742,6 +742,7 @@
#dynamic_library_path = '$libdir'
#gin_fuzzy_search_limit = 0
+#libpq_compression = on
#------------------------------------------------------------------------------
# LOCK MANAGEMENT
diff --git a/src/common/meson.build b/src/common/meson.build
index 1c9b8a3a018..c6b3bae6b54 100644
--- a/src/common/meson.build
+++ b/src/common/meson.build
@@ -30,6 +30,8 @@ common_sources = files(
'username.c',
'wait_error.c',
'wchar.c',
+ 'z_stream.c',
+ 'zpq_stream.c',
)
if ssl.found()
@@ -132,7 +134,7 @@ pgcommon_variants = {
'_shlib': default_lib_args + {
'pic': true,
'sources': common_sources_frontend_shlib,
- 'dependencies': [frontend_common_code],
+ 'dependencies': [frontend_common_code, lz4],
},
}
--
2.25.1
* Re: libpq compression (part 2)
@ 2022-11-17 00:27 Andrey Borodin <amborodin86@gmail.com>
parent: Justin Pryzby <pryzby@telsasoft.com>
0 siblings, 1 reply; 27+ messages in thread
From: Andrey Borodin @ 2022-11-17 00:27 UTC (permalink / raw)
To: Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On Tue, Nov 15, 2022 at 7:17 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
>
Also I've found one more TODO item for the patch. Currently in
fe-connect.c patch is doing buffer reset:
conn->inStart = conn->inCursor = conn->inEnd = 0;
This effectively consumes butes up tu current cursor. However, packet
inspection is continued. The patch works because in most cases
following code will issue re-read of message. Coincidentally.
/* Get the type of request. */
if (pqGetInt((int *) &areq, 4, conn))
{
/* We'll come back when there are more data */
return PGRES_POLLING_READING;
}
But I think we need a proper
goto keep_going;
to start from the beginning of the message.
Thank you!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-17 15:09 Peter Eisentraut <peter.eisentraut@enterprisedb.com>
parent: Andrey Borodin <amborodin86@gmail.com>
0 siblings, 1 reply; 27+ messages in thread
From: Peter Eisentraut @ 2022-11-17 15:09 UTC (permalink / raw)
To: Andrey Borodin <amborodin86@gmail.com>; Justin Pryzby <pryzby@telsasoft.com>; +Cc: Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On 17.11.22 01:27, Andrey Borodin wrote:
> Also I've found one more TODO item for the patch. Currently in
> fe-connect.c patch is doing buffer reset:
> conn->inStart = conn->inCursor = conn->inEnd = 0;
> This effectively consumes butes up tu current cursor. However, packet
> inspection is continued. The patch works because in most cases
> following code will issue re-read of message. Coincidentally.
> /* Get the type of request. */
> if (pqGetInt((int *) &areq, 4, conn))
> {
> /* We'll come back when there are more data */
> return PGRES_POLLING_READING;
> }
Note that the above code was just changed in dce92e59b1. I don't know
how that affects this patch set.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-18 01:07 Andrey Borodin <amborodin86@gmail.com>
parent: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
0 siblings, 1 reply; 27+ messages in thread
From: Andrey Borodin @ 2022-11-18 01:07 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On Thu, Nov 17, 2022 at 7:09 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
>
> Note that the above code was just changed in dce92e59b1.
Thanks!
> I don't know
> how that affects this patch set.
With dce92e59b1 it would be much easier to find a bug in the compression patch.
Some more notes about the patch. (sorry for posting review notes in so
many different messages)
1. zs_is_valid_impl_id(unsigned int id)
{
return id >= 0 && id < lengthof(zs_algorithms);
}
id is unsigned, no need to check it's non-negative.
2. This literal
{no_compression_name}
should be replaced by explicit form
{no_compression_name, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL}
3. Comments like "Return true if should, false if should not." are useless.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2022-11-18 16:31 Peter Eisentraut <peter.eisentraut@enterprisedb.com>
parent: Andrey Borodin <amborodin86@gmail.com>
0 siblings, 1 reply; 27+ messages in thread
From: Peter Eisentraut @ 2022-11-18 16:31 UTC (permalink / raw)
To: Andrey Borodin <amborodin86@gmail.com>; +Cc: Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
On 18.11.22 02:07, Andrey Borodin wrote:
> 2. This literal
> {no_compression_name}
> should be replaced by explicit form
> {no_compression_name, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL}
That doesn't seem better.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2023-08-10 14:47 Jonah H. Harris <jonah.harris@gmail.com>
parent: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
0 siblings, 1 reply; 27+ messages in thread
From: Jonah H. Harris @ 2023-08-10 14:47 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; +Cc: Andrey Borodin <amborodin86@gmail.com>; Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org, Andrey Borodin <x4mmm@yandex-team.ru>
Pinging to see if anyone has continued to work on this behind-the-scenes or
whether this is the latest patch set there is.
--
Jonah H. Harris
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: libpq compression (part 2)
@ 2023-08-11 11:31 Andrey M. Borodin <x4mmm@yandex-team.ru>
parent: Jonah H. Harris <jonah.harris@gmail.com>
0 siblings, 0 replies; 27+ messages in thread
From: Andrey M. Borodin @ 2023-08-11 11:31 UTC (permalink / raw)
To: Jonah H. Harris <jonah.harris@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>; Andrey Borodin <amborodin86@gmail.com>; Justin Pryzby <pryzby@telsasoft.com>; Jacob Champion <jchampion@timescale.com>; Daniil Zakhlystov <usernamedt@yandex-team.ru>; pgsql-hackers@lists.postgresql.org
> On 10 Aug 2023, at 19:47, Jonah H. Harris <jonah.harris@gmail.com> wrote:
>
> Pinging to see if anyone has continued to work on this behind-the-scenes or whether this is the latest patch set there is.
It's still on my TODO list, but I haven't done much review cycles yet. And the patch series already needs heavy rebase.
Thank for your interest in the topic.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 27+ messages in thread
end of thread, other threads:[~2023-08-11 11:31 UTC | newest]
Thread overview: 27+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-12-31 09:26 libpq compression (part 2) Daniil Zakhlystov <usernamedt@yandex-team.ru>
2022-01-01 17:25 ` Justin Pryzby <pryzby@telsasoft.com>
2022-01-03 11:16 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-01-07 18:46 ` Bruce Momjian <bruce@momjian.us>
2022-01-07 19:21 ` Justin Pryzby <pryzby@telsasoft.com>
2022-01-07 20:56 ` Stephen Frost <sfrost@snowman.net>
2022-01-07 21:09 ` Bruce Momjian <bruce@momjian.us>
2022-01-07 21:17 ` Stephen Frost <sfrost@snowman.net>
2022-01-12 09:38 ` Andrey Borodin <x4mmm@yandex-team.ru>
2022-01-12 14:15 ` Justin Pryzby <pryzby@telsasoft.com>
2022-01-13 21:12 ` Daniil Zakhlystov <usernamedt@yandex-team.ru>
2022-01-13 23:58 ` Justin Pryzby <pryzby@telsasoft.com>
2022-01-17 21:06 ` Daniil Zakhlystov <usernamedt@yandex-team.ru>
2022-01-18 04:39 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-02 21:33 ` Justin Pryzby <pryzby@telsasoft.com>
2022-03-03 08:50 ` Daniil Zakhlystov <usernamedt@yandex-team.ru>
2022-08-02 20:53 ` Jacob Champion <jchampion@timescale.com>
2022-11-12 21:47 ` Andrey Borodin <amborodin86@gmail.com>
2022-11-13 04:04 ` Andrey Borodin <amborodin86@gmail.com>
2022-11-15 03:44 ` Andrey Borodin <amborodin86@gmail.com>
2022-11-16 03:17 ` Justin Pryzby <pryzby@telsasoft.com>
2022-11-17 00:27 ` Andrey Borodin <amborodin86@gmail.com>
2022-11-17 15:09 ` Peter Eisentraut <peter.eisentraut@enterprisedb.com>
2022-11-18 01:07 ` Andrey Borodin <amborodin86@gmail.com>
2022-11-18 16:31 ` Peter Eisentraut <peter.eisentraut@enterprisedb.com>
2023-08-10 14:47 ` Jonah H. Harris <jonah.harris@gmail.com>
2023-08-11 11:31 ` Andrey M. Borodin <x4mmm@yandex-team.ru>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox