public inbox for [email protected]help / color / mirror / Atom feed
test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) 9+ messages / 3 participants [nested] [flat]
* test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 00:37 Michael Paquier <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Michael Paquier @ 2026-04-13 00:37 UTC (permalink / raw) To: Postgres hackers <[email protected]>; +Cc: Andrew Dunstan <[email protected]> Hi all, (Andrew in CC.) While reading Andrew's commit 2b5ba2a0a141, I was a bit sad to not see tests for these problems with pglz, applied with the fix down to v14. Relying on fuzzing is not really cool, because these consume resources and they may not even hit the correct target, and we want a maximum of deterministic tests. And then, I got reminded that one of my pet plugins does something close to that (used that around 9.5 for some FPW compression benchmarks): https://github.com/michaelpq/pg_plugins/tree/main/compress_test With this infrastructure already at hand, implementing the problematic tests with corrupted varlenas was a matter of minutes, leading me to the attached patch (bonus points for check_comprete and rawsize). I would like to apply that down to v14, like the previous commit that has fixed these cases with pglz. That should come in handy in case more bugs pop in this area of the code, especially with more compression methods in mind. Any objections and/or comments about that? -- Michael From 39eb787808d77d3cad30dc3d5ce2d02503326cba Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Mon, 13 Apr 2026 09:33:30 +0900 Subject: [PATCH] test_compression: Test module for compression methods The goal of this module is to provide tests for low-level APIs of compression methods. pglz is covered in this commit. This module includes also tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=false/true. Backpatch-through: 14 --- src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + src/test/modules/test_compression/.gitignore | 4 + src/test/modules/test_compression/Makefile | 23 +++++ .../expected/test_compression.out | 51 +++++++++++ src/test/modules/test_compression/meson.build | 33 +++++++ .../test_compression/sql/test_compression.sql | 36 ++++++++ .../test_compression--1.0.sql | 12 +++ .../test_compression/test_compression.c | 85 +++++++++++++++++++ .../test_compression/test_compression.control | 4 + 10 files changed, 250 insertions(+) create mode 100644 src/test/modules/test_compression/.gitignore create mode 100644 src/test/modules/test_compression/Makefile create mode 100644 src/test/modules/test_compression/expected/test_compression.out create mode 100644 src/test/modules/test_compression/meson.build create mode 100644 src/test/modules/test_compression/sql/test_compression.sql create mode 100644 src/test/modules/test_compression/test_compression--1.0.sql create mode 100644 src/test/modules/test_compression/test_compression.c create mode 100644 src/test/modules/test_compression/test_compression.control diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 0a74ab5c86f5..dbcd432f8c86 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -22,6 +22,7 @@ SUBDIRS = \ test_bloomfilter \ test_cloexec \ test_checksums \ + test_compression \ test_copy_callbacks \ test_custom_rmgrs \ test_custom_stats \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb3706..7cb26400d435 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -22,6 +22,7 @@ subdir('test_bitmapset') subdir('test_bloomfilter') subdir('test_cloexec') subdir('test_checksums') +subdir('test_compression') subdir('test_copy_callbacks') subdir('test_cplusplusext') subdir('test_custom_rmgrs') diff --git a/src/test/modules/test_compression/.gitignore b/src/test/modules/test_compression/.gitignore new file mode 100644 index 000000000000..5dcb3ff97235 --- /dev/null +++ b/src/test/modules/test_compression/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_compression/Makefile b/src/test/modules/test_compression/Makefile new file mode 100644 index 000000000000..82c6ace4dc8a --- /dev/null +++ b/src/test/modules/test_compression/Makefile @@ -0,0 +1,23 @@ +# src/test/modules/test_compression/Makefile + +MODULE_big = test_compression +OBJS = \ + $(WIN32RES) \ + test_compression.o +PGFILEDESC = "test_compression - test code for compression methods" + +EXTENSION = test_compression +DATA = test_compression--1.0.sql + +REGRESS = test_compression + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_compression/expected/test_compression.out b/src/test/modules/test_compression/expected/test_compression.out new file mode 100644 index 000000000000..837acc85dd49 --- /dev/null +++ b/src/test/modules/test_compression/expected/test_compression.out @@ -0,0 +1,51 @@ +CREATE EXTENSION test_compression; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; + ctrl_only_len +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/meson.build b/src/test/modules/test_compression/meson.build new file mode 100644 index 000000000000..b25144ce71cd --- /dev/null +++ b/src/test/modules/test_compression/meson.build @@ -0,0 +1,33 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +test_compression_sources = files( + 'test_compression.c', +) + +if host_system == 'windows' + test_compression_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_compression', + '--FILEDESC', 'test_compression - test code for compression methods',]) +endif + +test_compression = shared_module('test_compression', + test_compression_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_compression + +test_install_data += files( + 'test_compression.control', + 'test_compression--1.0.sql', +) + +tests += { + 'name': 'test_compression', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'regress': { + 'sql': [ + 'test_compression', + ], + }, +} diff --git a/src/test/modules/test_compression/sql/test_compression.sql b/src/test/modules/test_compression/sql/test_compression.sql new file mode 100644 index 000000000000..4775b5ab582e --- /dev/null +++ b/src/test/modules/test_compression/sql/test_compression.sql @@ -0,0 +1,36 @@ +CREATE EXTENSION test_compression; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); + +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/test_compression--1.0.sql b/src/test/modules/test_compression/test_compression--1.0.sql new file mode 100644 index 000000000000..cc789df87340 --- /dev/null +++ b/src/test/modules/test_compression/test_compression--1.0.sql @@ -0,0 +1,12 @@ +/* src/test/modules/test_compression/test_compression--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION test_compression" to load this file. \quit + +CREATE FUNCTION test_pglz_compress(bytea) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; + +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; diff --git a/src/test/modules/test_compression/test_compression.c b/src/test/modules/test_compression/test_compression.c new file mode 100644 index 000000000000..2a1d27999395 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.c @@ -0,0 +1,85 @@ +/*-------------------------------------------------------------------------- + * + * test_compression.c + * Test harness for compression methods. + * + * Copyright (c) 2026, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/test/modules/test_compression/test_compression.c + * + * ------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "varatt.h" + +PG_MODULE_MAGIC; + +PG_FUNCTION_INFO_V1(test_pglz_compress); +PG_FUNCTION_INFO_V1(test_pglz_decompress); + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + int32 maxout = PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result = (bytea *) palloc(maxout + VARHDRSZ); + clen = pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fails. + */ +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + int32 rawsize = PG_GETARG_INT32(1); + bool check_complete = PG_GETARG_BOOL(2); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result = (bytea *) palloc(rawsize + VARHDRSZ); + + dlen = pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/modules/test_compression/test_compression.control b/src/test/modules/test_compression/test_compression.control new file mode 100644 index 000000000000..f707d4dfcf51 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.control @@ -0,0 +1,4 @@ +comment = 'Test code for compression methods' +default_version = '1.0' +module_pathname = '$libdir/test_compression' +relocatable = true -- 2.53.0 Attachments: [text/plain] 0001-test_compression-Test-module-for-compression-methods.patch (12.3K, 2-0001-test_compression-Test-module-for-compression-methods.patch) download | inline diff: From 39eb787808d77d3cad30dc3d5ce2d02503326cba Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Mon, 13 Apr 2026 09:33:30 +0900 Subject: [PATCH] test_compression: Test module for compression methods The goal of this module is to provide tests for low-level APIs of compression methods. pglz is covered in this commit. This module includes also tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=false/true. Backpatch-through: 14 --- src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + src/test/modules/test_compression/.gitignore | 4 + src/test/modules/test_compression/Makefile | 23 +++++ .../expected/test_compression.out | 51 +++++++++++ src/test/modules/test_compression/meson.build | 33 +++++++ .../test_compression/sql/test_compression.sql | 36 ++++++++ .../test_compression--1.0.sql | 12 +++ .../test_compression/test_compression.c | 85 +++++++++++++++++++ .../test_compression/test_compression.control | 4 + 10 files changed, 250 insertions(+) create mode 100644 src/test/modules/test_compression/.gitignore create mode 100644 src/test/modules/test_compression/Makefile create mode 100644 src/test/modules/test_compression/expected/test_compression.out create mode 100644 src/test/modules/test_compression/meson.build create mode 100644 src/test/modules/test_compression/sql/test_compression.sql create mode 100644 src/test/modules/test_compression/test_compression--1.0.sql create mode 100644 src/test/modules/test_compression/test_compression.c create mode 100644 src/test/modules/test_compression/test_compression.control diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 0a74ab5c86f5..dbcd432f8c86 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -22,6 +22,7 @@ SUBDIRS = \ test_bloomfilter \ test_cloexec \ test_checksums \ + test_compression \ test_copy_callbacks \ test_custom_rmgrs \ test_custom_stats \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb3706..7cb26400d435 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -22,6 +22,7 @@ subdir('test_bitmapset') subdir('test_bloomfilter') subdir('test_cloexec') subdir('test_checksums') +subdir('test_compression') subdir('test_copy_callbacks') subdir('test_cplusplusext') subdir('test_custom_rmgrs') diff --git a/src/test/modules/test_compression/.gitignore b/src/test/modules/test_compression/.gitignore new file mode 100644 index 000000000000..5dcb3ff97235 --- /dev/null +++ b/src/test/modules/test_compression/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_compression/Makefile b/src/test/modules/test_compression/Makefile new file mode 100644 index 000000000000..82c6ace4dc8a --- /dev/null +++ b/src/test/modules/test_compression/Makefile @@ -0,0 +1,23 @@ +# src/test/modules/test_compression/Makefile + +MODULE_big = test_compression +OBJS = \ + $(WIN32RES) \ + test_compression.o +PGFILEDESC = "test_compression - test code for compression methods" + +EXTENSION = test_compression +DATA = test_compression--1.0.sql + +REGRESS = test_compression + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_compression/expected/test_compression.out b/src/test/modules/test_compression/expected/test_compression.out new file mode 100644 index 000000000000..837acc85dd49 --- /dev/null +++ b/src/test/modules/test_compression/expected/test_compression.out @@ -0,0 +1,51 @@ +CREATE EXTENSION test_compression; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; + ctrl_only_len +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/meson.build b/src/test/modules/test_compression/meson.build new file mode 100644 index 000000000000..b25144ce71cd --- /dev/null +++ b/src/test/modules/test_compression/meson.build @@ -0,0 +1,33 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +test_compression_sources = files( + 'test_compression.c', +) + +if host_system == 'windows' + test_compression_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_compression', + '--FILEDESC', 'test_compression - test code for compression methods',]) +endif + +test_compression = shared_module('test_compression', + test_compression_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_compression + +test_install_data += files( + 'test_compression.control', + 'test_compression--1.0.sql', +) + +tests += { + 'name': 'test_compression', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'regress': { + 'sql': [ + 'test_compression', + ], + }, +} diff --git a/src/test/modules/test_compression/sql/test_compression.sql b/src/test/modules/test_compression/sql/test_compression.sql new file mode 100644 index 000000000000..4775b5ab582e --- /dev/null +++ b/src/test/modules/test_compression/sql/test_compression.sql @@ -0,0 +1,36 @@ +CREATE EXTENSION test_compression; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); + +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/test_compression--1.0.sql b/src/test/modules/test_compression/test_compression--1.0.sql new file mode 100644 index 000000000000..cc789df87340 --- /dev/null +++ b/src/test/modules/test_compression/test_compression--1.0.sql @@ -0,0 +1,12 @@ +/* src/test/modules/test_compression/test_compression--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION test_compression" to load this file. \quit + +CREATE FUNCTION test_pglz_compress(bytea) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; + +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; diff --git a/src/test/modules/test_compression/test_compression.c b/src/test/modules/test_compression/test_compression.c new file mode 100644 index 000000000000..2a1d27999395 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.c @@ -0,0 +1,85 @@ +/*-------------------------------------------------------------------------- + * + * test_compression.c + * Test harness for compression methods. + * + * Copyright (c) 2026, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/test/modules/test_compression/test_compression.c + * + * ------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "varatt.h" + +PG_MODULE_MAGIC; + +PG_FUNCTION_INFO_V1(test_pglz_compress); +PG_FUNCTION_INFO_V1(test_pglz_decompress); + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + int32 maxout = PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result = (bytea *) palloc(maxout + VARHDRSZ); + clen = pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fails. + */ +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + int32 rawsize = PG_GETARG_INT32(1); + bool check_complete = PG_GETARG_BOOL(2); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result = (bytea *) palloc(rawsize + VARHDRSZ); + + dlen = pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/modules/test_compression/test_compression.control b/src/test/modules/test_compression/test_compression.control new file mode 100644 index 000000000000..f707d4dfcf51 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.control @@ -0,0 +1,4 @@ +comment = 'Test code for compression methods' +default_version = '1.0' +module_pathname = '$libdir/test_compression' +relocatable = true -- 2.53.0 [application/pgp-signature] signature.asc (833B, 3-signature.asc) download ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 02:20 Andres Freund <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Andres Freund @ 2026-04-13 02:20 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> Hi, On 2026-04-13 09:37:55 +0900, Michael Paquier wrote: > With this infrastructure already at hand, implementing the > problematic tests with corrupted varlenas was a matter of minutes, > leading me to the attached patch (bonus points for check_comprete and > rawsize) I think it doesn't scale to have a whole postgres cluster for a test that takes a few milliseconds. The amount of IO one run of all of postgres' tests is doing is getting unmangeable, and lots of clusters that are used for a a second or two that are immediately destroyed contributes substantially to that. One PG_TEST_NOCLEAN=1 run with meson ends up with a 33GB testrun/ directory. And that's without even counting all the pg_regress tests, because there's no convenient way to disable that. On a smaller machine much of that will be written to disk due to cache/memory pressure. There's really no reason for something like this to be a test doing tests via SQL from what I can tell. If it does not to be via SSL, can we please start to find a way to combine tiny stuff like this? We're working hard at making our tests grow unsustainable. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 03:52 Michael Paquier <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Michael Paquier @ 2026-04-13 03:52 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> On Sun, Apr 12, 2026 at 10:20:43PM -0400, Andres Freund wrote: > There's really no reason for something like this to be a test doing tests via > SQL from what I can tell. > > If it does not to be via SSL, can we please start to find a way to combine > tiny stuff like this? We're working hard at making our tests grow > unsustainable. If we care about `make check` rather than `make installcheck`, it seems to me that a solution already exists in the shape of C function called through the main regression test suite. And for the specific case of this thread, I could live with two new functions in regress.c that are then called in compression.sql. Would this idea work for you when it comes to this proposal? -- Michael Attachments: [application/pgp-signature] signature.asc (833B, 2-signature.asc) download ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 14:08 Andres Freund <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 2 replies; 9+ messages in thread From: Andres Freund @ 2026-04-13 14:08 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> Hi, On 2026-04-13 12:52:59 +0900, Michael Paquier wrote: > On Sun, Apr 12, 2026 at 10:20:43PM -0400, Andres Freund wrote: > > There's really no reason for something like this to be a test doing tests via > > SQL from what I can tell. > > > > If it does not to be via SSL, can we please start to find a way to combine > > tiny stuff like this? We're working hard at making our tests grow > > unsustainable. > > If we care about `make check` rather than `make installcheck` I don't really see the difference WRT that in this case? installcheck still has regress.so available, no? > it seems to me that a solution already exists in the shape of C function > called through the main regression test suite. I guess that'd work. I don't really see the point in calling these via SQL, tbh, but if that's easier for you, using a regress.c helper woul do the trick. > Would this idea work for you when it comes to this proposal? Yes. I think we need to combine about half the modules in src/test/modules, the current course is absurd: 16: 37 17: 46 18: 49 dev: 62 [Almost] All of them create a new initdb'd cluster. ~50MB of writes for a short test is insane. Doing low-level unittests by doing inter-process communication from psql to the server, marshalling everything through text, is insane. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 16:06 Jacob Champion <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 0 replies; 9+ messages in thread From: Jacob Champion @ 2026-04-13 16:06 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Michael Paquier <[email protected]>; Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> On Mon, Apr 13, 2026 at 7:08 AM Andres Freund <[email protected]> wrote: > > Doing low-level unittests by doing inter-process > communication from psql to the server, marshalling everything through text, is > insane. +1. (I'm more than happy to provide eyes/code/pairing/etc. for C unit test infrastructure, since I've been pushing on that from the client side recently.) --Jacob ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 17:13 Michael Paquier <[email protected]> parent: Andres Freund <[email protected]> 1 sibling, 1 reply; 9+ messages in thread From: Michael Paquier @ 2026-04-13 17:13 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> On Mon, Apr 13, 2026 at 10:08:16AM -0400, Andres Freund wrote: > I guess that'd work. I don't really see the point in calling these via SQL, > tbh, but if that's easier for you, using a regress.c helper woul do the trick. Okay, thanks for the opinion. It makes the addition of more tests slightly easier as there is no need to rely on an extra wrapper of what's in pg_lzcompress.h. I have done that in the attached, with additions to the main regression test suite. -- Michael From ad27ede8a798c264de961b8bc37205a2635e43c5 Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Tue, 14 Apr 2026 02:01:26 +0900 Subject: [PATCH v2] Add tests for low-level PGLZ compression The goal of this module is to provide an entry point for the coverage of for low-level APIs of compression and decompression of PGLZ. This includes tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=false/true. Backpatch-through: 14 --- .../regress/expected/compression_pglz.out | 62 +++++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/regress.c | 66 +++++++++++++++++++ src/test/regress/sql/compression_pglz.sql | 49 ++++++++++++++ 4 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 src/test/regress/expected/compression_pglz.out create mode 100644 src/test/regress/sql/compression_pglz.sql diff --git a/src/test/regress/expected/compression_pglz.out b/src/test/regress/expected/compression_pglz.out new file mode 100644 index 000000000000..daf457f4ac0a --- /dev/null +++ b/src/test/regress/expected/compression_pglz.out @@ -0,0 +1,62 @@ +-- +-- Tests for PGLZ compression +-- +-- directory paths and dlsuffix are passed to us in environment variables +\getenv libdir PG_LIBDIR +\getenv dlsuffix PG_DLSUFFIX +\set regresslib :libdir '/regress' :dlsuffix +CREATE FUNCTION test_pglz_compress(bytea) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; + ctrl_only_len +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index cc365393bb7d..3b059906161f 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -128,7 +128,7 @@ test: partition_merge partition_split partition_join partition_prune reloptions # event_trigger depends on create_am and cannot run concurrently with # any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage -test: oidjoins event_trigger +test: oidjoins event_trigger compression_pglz test: role_ddl tablespace_ddl database_ddl diff --git a/src/test/regress/regress.c b/src/test/regress/regress.c index 5c19f70b6e8e..2bcb5559a452 100644 --- a/src/test/regress/regress.c +++ b/src/test/regress/regress.c @@ -27,6 +27,7 @@ #include "catalog/pg_type.h" #include "commands/sequence.h" #include "commands/trigger.h" +#include "common/pg_lzcompress.h" #include "executor/executor.h" #include "executor/functions.h" #include "executor/spi.h" @@ -1422,3 +1423,68 @@ test_instr_time(PG_FUNCTION_ARGS) PG_RETURN_BOOL(true); } + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +PG_FUNCTION_INFO_V1(test_pglz_compress); +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + int32 maxout = PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result = (bytea *) palloc(maxout + VARHDRSZ); + clen = pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fails. + */ +PG_FUNCTION_INFO_V1(test_pglz_decompress); +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + int32 rawsize = PG_GETARG_INT32(1); + bool check_complete = PG_GETARG_BOOL(2); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result = (bytea *) palloc(rawsize + VARHDRSZ); + + dlen = pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/regress/sql/compression_pglz.sql b/src/test/regress/sql/compression_pglz.sql new file mode 100644 index 000000000000..115f1300d748 --- /dev/null +++ b/src/test/regress/sql/compression_pglz.sql @@ -0,0 +1,49 @@ +-- +-- Tests for PGLZ compression +-- + +-- directory paths and dlsuffix are passed to us in environment variables +\getenv libdir PG_LIBDIR +\getenv dlsuffix PG_DLSUFFIX + +\set regresslib :libdir '/regress' :dlsuffix + +CREATE FUNCTION test_pglz_compress(bytea) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); -- 2.53.0 Attachments: [text/plain] v2-0001-Add-tests-for-low-level-PGLZ-compression.patch (8.5K, 2-v2-0001-Add-tests-for-low-level-PGLZ-compression.patch) download | inline diff: From ad27ede8a798c264de961b8bc37205a2635e43c5 Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Tue, 14 Apr 2026 02:01:26 +0900 Subject: [PATCH v2] Add tests for low-level PGLZ compression The goal of this module is to provide an entry point for the coverage of for low-level APIs of compression and decompression of PGLZ. This includes tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=false/true. Backpatch-through: 14 --- .../regress/expected/compression_pglz.out | 62 +++++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/regress.c | 66 +++++++++++++++++++ src/test/regress/sql/compression_pglz.sql | 49 ++++++++++++++ 4 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 src/test/regress/expected/compression_pglz.out create mode 100644 src/test/regress/sql/compression_pglz.sql diff --git a/src/test/regress/expected/compression_pglz.out b/src/test/regress/expected/compression_pglz.out new file mode 100644 index 000000000000..daf457f4ac0a --- /dev/null +++ b/src/test/regress/expected/compression_pglz.out @@ -0,0 +1,62 @@ +-- +-- Tests for PGLZ compression +-- +-- directory paths and dlsuffix are passed to us in environment variables +\getenv libdir PG_LIBDIR +\getenv dlsuffix PG_DLSUFFIX +\set regresslib :libdir '/regress' :dlsuffix +CREATE FUNCTION test_pglz_compress(bytea) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; + ctrl_only_len +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index cc365393bb7d..3b059906161f 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -128,7 +128,7 @@ test: partition_merge partition_split partition_join partition_prune reloptions # event_trigger depends on create_am and cannot run concurrently with # any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage -test: oidjoins event_trigger +test: oidjoins event_trigger compression_pglz test: role_ddl tablespace_ddl database_ddl diff --git a/src/test/regress/regress.c b/src/test/regress/regress.c index 5c19f70b6e8e..2bcb5559a452 100644 --- a/src/test/regress/regress.c +++ b/src/test/regress/regress.c @@ -27,6 +27,7 @@ #include "catalog/pg_type.h" #include "commands/sequence.h" #include "commands/trigger.h" +#include "common/pg_lzcompress.h" #include "executor/executor.h" #include "executor/functions.h" #include "executor/spi.h" @@ -1422,3 +1423,68 @@ test_instr_time(PG_FUNCTION_ARGS) PG_RETURN_BOOL(true); } + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +PG_FUNCTION_INFO_V1(test_pglz_compress); +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + int32 maxout = PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result = (bytea *) palloc(maxout + VARHDRSZ); + clen = pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fails. + */ +PG_FUNCTION_INFO_V1(test_pglz_decompress); +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + int32 rawsize = PG_GETARG_INT32(1); + bool check_complete = PG_GETARG_BOOL(2); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result = (bytea *) palloc(rawsize + VARHDRSZ); + + dlen = pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/regress/sql/compression_pglz.sql b/src/test/regress/sql/compression_pglz.sql new file mode 100644 index 000000000000..115f1300d748 --- /dev/null +++ b/src/test/regress/sql/compression_pglz.sql @@ -0,0 +1,49 @@ +-- +-- Tests for PGLZ compression +-- + +-- directory paths and dlsuffix are passed to us in environment variables +\getenv libdir PG_LIBDIR +\getenv dlsuffix PG_DLSUFFIX + +\set regresslib :libdir '/regress' :dlsuffix + +CREATE FUNCTION test_pglz_compress(bytea) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) + RETURNS bytea + AS :'regresslib' LANGUAGE C STRICT; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); -- 2.53.0 [application/pgp-signature] signature.asc (833B, 3-signature.asc) download ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 18:50 Andres Freund <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Andres Freund @ 2026-04-13 18:50 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> Hi, On 2026-04-14 02:13:52 +0900, Michael Paquier wrote: > +++ b/src/test/regress/parallel_schedule > @@ -128,7 +128,7 @@ test: partition_merge partition_split partition_join partition_prune reloptions > # event_trigger depends on create_am and cannot run concurrently with > # any test that runs DDL > # oidjoins is read-only, though, and should run late for best coverage > -test: oidjoins event_trigger > +test: oidjoins event_trigger compression_pglz The new test creates a function, so I don't think this will be safe? Greetings, Andres Freund ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-13 19:01 Michael Paquier <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Michael Paquier @ 2026-04-13 19:01 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> On Mon, Apr 13, 2026 at 02:50:31PM -0400, Andres Freund wrote: > Hi, > > On 2026-04-14 02:13:52 +0900, Michael Paquier wrote: > > +++ b/src/test/regress/parallel_schedule >> @@ -128,7 +128,7 @@ test: partition_merge partition_split partition_join partition_prune reloptions >> # event_trigger depends on create_am and cannot run concurrently with >> # any test that runs DDL >> # oidjoins is read-only, though, and should run late for best coverage >> -test: oidjoins event_trigger >> +test: oidjoins event_trigger compression_pglz > > The new test creates a function, so I don't think this will be safe? Yep, that was a thinko. The group for plancache still has room for one test, as it can go up to 19, so we could move one test in the group of partition_merge t the group of plancache, keeping all the compression tests together. Anyway, how about just creating a new group for all the compression tests? I have a patch set in preparation for zstd and TOAST, where I would put the new test suite in this group. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, 2-signature.asc) download ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) @ 2026-04-14 21:16 Michael Paquier <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: Michael Paquier @ 2026-04-14 21:16 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Postgres hackers <[email protected]>; Andrew Dunstan <[email protected]> On Tue, Apr 14, 2026 at 04:01:53AM +0900, Michael Paquier wrote: > Anyway, how about just creating a new group for all the compression > tests? I have a patch set in preparation for zstd and TOAST, where I > would put the new test suite in this group. At the end I have used a new group. For branches older than v18, the parallel groups close by had a lot more room, but I have given priority to consistency with HEAD in parallel_schedule across the board. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, 2-signature.asc) download ^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2026-04-14 21:16 UTC | newest] Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-04-13 00:37 test_compression, test module for low-level compression APIs (for 2b5ba2a0a141) Michael Paquier <[email protected]> 2026-04-13 02:20 ` Andres Freund <[email protected]> 2026-04-13 03:52 ` Michael Paquier <[email protected]> 2026-04-13 14:08 ` Andres Freund <[email protected]> 2026-04-13 16:06 ` Jacob Champion <[email protected]> 2026-04-13 17:13 ` Michael Paquier <[email protected]> 2026-04-13 18:50 ` Andres Freund <[email protected]> 2026-04-13 19:01 ` Michael Paquier <[email protected]> 2026-04-14 21:16 ` Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox