Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wC5JW-001eFH-2p for pgsql-hackers@arkaria.postgresql.org; Mon, 13 Apr 2026 00:38:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1wC5JT-003eIP-0A for pgsql-hackers@arkaria.postgresql.org; Mon, 13 Apr 2026 00:38:03 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wC5JR-003eIG-2c for pgsql-hackers@lists.postgresql.org; Mon, 13 Apr 2026 00:38:03 +0000 Received: from fhigh-b6-smtp.messagingengine.com ([202.12.124.157]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wC5JP-00000000iKK-37lE for pgsql-hackers@lists.postgresql.org; Mon, 13 Apr 2026 00:38:01 +0000 Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfhigh.stl.internal (Postfix) with ESMTP id 1F8887A00A2; Sun, 12 Apr 2026 20:37:59 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Sun, 12 Apr 2026 20:37:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm2; t=1776040678; x=1776127078; bh=bKRSkgkzmyc6p3h8ib/eQgpbIU3GMS2k 1rxNqtwcO+M=; b=Gv9E+TleEyMdZX+l3Q/WzdOE2PR57Vk1X775laiQsBCVqG2C hs1skac9rx2snnBhYUekXyI81tIkTgft1yhApHsRoYaqto+THAwCHRTJmbiX25xX diOo0ivQewUdTAd3dw7lt5NaIJgp2f5w1j4gDgl4NdbYfttBjDqa5+Ivc2raC4Kg QPHQdEAYXP/d7JTCVC37pbaLj3DYJDZmwlUu55mYWXx8b9DkqIsbnqiglFh/Tmqg rQXmnCg4jHtGtWUWtjXgGJtTppSLArkHiZgXQn6n7wP/Z1FG3WU/qmtqK6b7QYPf yf6oks566M/x3YKhRL9rN+dQvCaZNbhG0rQLhQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1776040678; x= 1776127078; bh=bKRSkgkzmyc6p3h8ib/eQgpbIU3GMS2k1rxNqtwcO+M=; b=Q JE/AKWvXXc2UkcC91GpP7D+DTNTo16Dt2A/U/6106OEQ9RSg5/l9yYiHBUmHEYWs URhzzbFi4xImbhW2/SSe9I/TOodeaebSm6iuxFRiAgObFRjtMqIq55CYFic9ybuG q//IUIOWaCcg9xr9ZcVecbHimH9aR2DyOSubVgpNH0aPyWBFdAKlpbN/ZDoK3zM1 AbhAzGwer+Av7TMHLixL/d5RdbW5loTtY0mn5+LG8OqXQ3wtcHxJn6NpJv0om8tv pzmyILB0Nt4XXQn/igbPz8GEWAZ2V80xmIBSCikL9/aIhNyyIIGQQ8s3A2pm9qAF fuYOLCeNiwemoDruy6lcA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefhedrtddtgdefieejlecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenfghrlh cuvffnffculdefhedmnecujfgurhepfffhvfevuffkgggtugesghdtreertddtvdenucfh rhhomhepofhitghhrggvlhcurfgrqhhuihgvrhcuoehmihgthhgrvghlsehprghquhhivg hrrdighiiiqeenucggtffrrghtthgvrhhnpedtleehtdeihfegueeiueeghfekgeelueef geekudegudeuleejueetudegtedvjeenucffohhmrghinhepghhithhhuhgsrdgtohhmne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhhitghh rggvlhesphgrqhhuihgvrhdrgiihiidpnhgspghrtghpthhtohepvddpmhhouggvpehsmh htphhouhhtpdhrtghpthhtohepphhgshhqlhdqhhgrtghkvghrsheslhhishhtshdrphho shhtghhrvghsqhhlrdhorhhgpdhrtghpthhtoheprghnughrvgifseguuhhnshhlrghnvg drnhgvth X-ME-Proxy: Feedback-ID: i0fe9450f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 12 Apr 2026 20:37:58 -0400 (EDT) Date: Mon, 13 Apr 2026 09:37:55 +0900 From: Michael Paquier To: Postgres hackers Cc: Andrew Dunstan Subject: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="X/5kqGmDxNVhcJlq" Content-Disposition: inline List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --X/5kqGmDxNVhcJlq Content-Type: multipart/mixed; boundary="7xw0zRi8dB+Sc2rT" Content-Disposition: inline --7xw0zRi8dB+Sc2rT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi all, (Andrew in CC.) While reading Andrew's commit 2b5ba2a0a141, I was a bit sad to not see tests for these problems with pglz, applied with the fix down to v14. Relying on fuzzing is not really cool, because these consume resources and they may not even hit the correct target, and we want a maximum of deterministic tests. And then, I got reminded that one of my pet plugins does something close to that (used that around 9.5 for some FPW compression benchmarks): https://github.com/michaelpq/pg_plugins/tree/main/compress_test With this infrastructure already at hand, implementing the problematic tests with corrupted varlenas was a matter of minutes, leading me to the attached patch (bonus points for check_comprete and rawsize). I would like to apply that down to v14, like the previous commit that has fixed these cases with pglz. That should come in handy in case more bugs pop in this area of the code, especially with more compression methods in mind. Any objections and/or comments about that? -- Michael --7xw0zRi8dB+Sc2rT Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=0001-test_compression-Test-module-for-compression-methods.patch Content-Transfer-Encoding: quoted-printable =46rom 39eb787808d77d3cad30dc3d5ce2d02503326cba Mon Sep 17 00:00:00 2001 =46rom: Michael Paquier Date: Mon, 13 Apr 2026 09:33:30 +0900 Subject: [PATCH] test_compression: Test module for compression methods The goal of this module is to provide tests for low-level APIs of compression methods. pglz is covered in this commit. This module includes also tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=3D18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=3Dfalse/true. Backpatch-through: 14 --- src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + src/test/modules/test_compression/.gitignore | 4 + src/test/modules/test_compression/Makefile | 23 +++++ .../expected/test_compression.out | 51 +++++++++++ src/test/modules/test_compression/meson.build | 33 +++++++ .../test_compression/sql/test_compression.sql | 36 ++++++++ .../test_compression--1.0.sql | 12 +++ .../test_compression/test_compression.c | 85 +++++++++++++++++++ .../test_compression/test_compression.control | 4 + 10 files changed, 250 insertions(+) create mode 100644 src/test/modules/test_compression/.gitignore create mode 100644 src/test/modules/test_compression/Makefile create mode 100644 src/test/modules/test_compression/expected/test_compres= sion.out create mode 100644 src/test/modules/test_compression/meson.build create mode 100644 src/test/modules/test_compression/sql/test_compression.= sql create mode 100644 src/test/modules/test_compression/test_compression--1.0= =2Esql create mode 100644 src/test/modules/test_compression/test_compression.c create mode 100644 src/test/modules/test_compression/test_compression.cont= rol diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 0a74ab5c86f5..dbcd432f8c86 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -22,6 +22,7 @@ SUBDIRS =3D \ test_bloomfilter \ test_cloexec \ test_checksums \ + test_compression \ test_copy_callbacks \ test_custom_rmgrs \ test_custom_stats \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb3706..7cb26400d435 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -22,6 +22,7 @@ subdir('test_bitmapset') subdir('test_bloomfilter') subdir('test_cloexec') subdir('test_checksums') +subdir('test_compression') subdir('test_copy_callbacks') subdir('test_cplusplusext') subdir('test_custom_rmgrs') diff --git a/src/test/modules/test_compression/.gitignore b/src/test/module= s/test_compression/.gitignore new file mode 100644 index 000000000000..5dcb3ff97235 --- /dev/null +++ b/src/test/modules/test_compression/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_compression/Makefile b/src/test/modules/= test_compression/Makefile new file mode 100644 index 000000000000..82c6ace4dc8a --- /dev/null +++ b/src/test/modules/test_compression/Makefile @@ -0,0 +1,23 @@ +# src/test/modules/test_compression/Makefile + +MODULE_big =3D test_compression +OBJS =3D \ + $(WIN32RES) \ + test_compression.o +PGFILEDESC =3D "test_compression - test code for compression methods" + +EXTENSION =3D test_compression +DATA =3D test_compression--1.0.sql + +REGRESS =3D test_compression + +ifdef USE_PGXS +PG_CONFIG =3D pg_config +PGXS :=3D $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir =3D src/test/modules/test_compression +top_builddir =3D ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_compression/expected/test_compression.ou= t b/src/test/modules/test_compression/expected/test_compression.out new file mode 100644 index 000000000000..837acc85dd49 --- /dev/null +++ b/src/test/modules/test_compression/expected/test_compression.out @@ -0,0 +1,51 @@ +CREATE EXTENSION test_compression; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) =3D + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok=20 +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) =3D + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok=20 +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_on= ly_len; + ctrl_only_len=20 +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=3D18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/meson.build b/src/test/modul= es/test_compression/meson.build new file mode 100644 index 000000000000..b25144ce71cd --- /dev/null +++ b/src/test/modules/test_compression/meson.build @@ -0,0 +1,33 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +test_compression_sources =3D files( + 'test_compression.c', +) + +if host_system =3D=3D 'windows' + test_compression_sources +=3D rc_lib_gen.process(win32ver_rc, extra_args= : [ + '--NAME', 'test_compression', + '--FILEDESC', 'test_compression - test code for compression methods',]) +endif + +test_compression =3D shared_module('test_compression', + test_compression_sources, + kwargs: pg_test_mod_args, +) +test_install_libs +=3D test_compression + +test_install_data +=3D files( + 'test_compression.control', + 'test_compression--1.0.sql', +) + +tests +=3D { + 'name': 'test_compression', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'regress': { + 'sql': [ + 'test_compression', + ], + }, +} diff --git a/src/test/modules/test_compression/sql/test_compression.sql b/s= rc/test/modules/test_compression/sql/test_compression.sql new file mode 100644 index 000000000000..4775b5ab582e --- /dev/null +++ b/src/test/modules/test_compression/sql/test_compression.sql @@ -0,0 +1,36 @@ +CREATE EXTENSION test_compression; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) =3D + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) =3D + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_on= ly_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=3D18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); + +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/test_compression--1.0.sql b/= src/test/modules/test_compression/test_compression--1.0.sql new file mode 100644 index 000000000000..cc789df87340 --- /dev/null +++ b/src/test/modules/test_compression/test_compression--1.0.sql @@ -0,0 +1,12 @@ +/* src/test/modules/test_compression/test_compression--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION test_compression" to load this file. \quit + +CREATE FUNCTION test_pglz_compress(bytea) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; + +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; diff --git a/src/test/modules/test_compression/test_compression.c b/src/tes= t/modules/test_compression/test_compression.c new file mode 100644 index 000000000000..2a1d27999395 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.c @@ -0,0 +1,85 @@ +/*------------------------------------------------------------------------= -- + * + * test_compression.c + * Test harness for compression methods. + * + * Copyright (c) 2026, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/test/modules/test_compression/test_compression.c + * + * -----------------------------------------------------------------------= -- + */ +#include "postgres.h" + +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "varatt.h" + +PG_MODULE_MAGIC; + +PG_FUNCTION_INFO_V1(test_pglz_compress); +PG_FUNCTION_INFO_V1(test_pglz_decompress); + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input =3D PG_GETARG_BYTEA_PP(0); + char *source =3D VARDATA_ANY(input); + int32 slen =3D VARSIZE_ANY_EXHDR(input); + int32 maxout =3D PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result =3D (bytea *) palloc(maxout + VARHDRSZ); + clen =3D pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fail= s. + */ +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input =3D PG_GETARG_BYTEA_PP(0); + int32 rawsize =3D PG_GETARG_INT32(1); + bool check_complete =3D PG_GETARG_BOOL(2); + char *source =3D VARDATA_ANY(input); + int32 slen =3D VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result =3D (bytea *) palloc(rawsize + VARHDRSZ); + + dlen =3D pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/modules/test_compression/test_compression.control b/s= rc/test/modules/test_compression/test_compression.control new file mode 100644 index 000000000000..f707d4dfcf51 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.control @@ -0,0 +1,4 @@ +comment =3D 'Test code for compression methods' +default_version =3D '1.0' +module_pathname =3D '$libdir/test_compression' +relocatable =3D true --=20 2.53.0 --7xw0zRi8dB+Sc2rT-- --X/5kqGmDxNVhcJlq Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmncOuMACgkQnvQgOdby QH3jbBAAh0VHhyQj4LhTtOMbCYlKW/3mpoz3e+gq7451OmkmIiVL/tXZJoAbaVBo l2HIxRULEYCK9CqPO2civGKm8KZyGPH9pNoApAZhrwno8jlfveSPFF3UefmsfUtm wVFYY3+d/xH3UOdAME0ZFigkqbx4tEvQavLXlD4iSzoKLcf34n22mwccp557Cur2 IyO2hB6JsdL75dI73EQQB+NXHY3TgXmDoqX0e7mu/x0v6MUtwmtBb007qTAwJQWR dIwQYbMBFoZht92p6SurZtV8fGr2v4Cgu0SfuXkdVqOiM1RMhmjJf9ieHAf06LwZ ftIRCkRGn5qRTBydrMM4Su8CrV6ku5iUypiIxTWpSSuJl1GNViqirCcBj+MJUIH3 I0vTaGBzq/JOoMWIXzTqXX61yFOMVVUzGMXtNh5BRob1OXlLgwaPi/EyirU1JdpS nkHpkXA0p7H1yx8vaclj5kLQvXuWOWa8OsgvL2IdDVTG314ozP+8+EWPuU6f4POW sclqTlbIGHdCx265Jduyl4UwPwLuAIFAQOVo2tVOxF6ozb/uxDTPx9LGOzXkOcID ss/hRCh7N/xj42k/IRyhftkAQJsKTqq2+/IgmQv611snxqCr3tR3zfKDmj+f8t/R NHoj4rEgrLbZ0/6s3EJHU1t0lXUpgW/n3gaC0gjGULP3b8uYsTg= =vz1g -----END PGP SIGNATURE----- --X/5kqGmDxNVhcJlq--