public inbox for [email protected]
help / color / mirror / Atom feedCompress prune/freeze records with Delta Frame of Reference algorithm
17+ messages / 4 participants
[nested] [flat]
* Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-03-07 13:56 Evgeny Voropaev <[email protected]>
0 siblings, 2 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-03-07 13:56 UTC (permalink / raw)
To: pgsql-hackers
Hi hackers,
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Currently, these
sequences are uncompressed, which adds overhead. Eliminating empty
leading bits in these offsets can reduce the size of the record, and
applying bit-packing algorithms can reduce it even further. Reducing
WAL record size also lessens the load on disk and network, which are
heavily used for storing WAL and transmitting it to replicas. For
example, with BLCKSZ equal to 8192, the maximum tuple offset is around
580. Despite this, all offsets are stored as 16-bit integers with no
compression applied.
In the proposed patch, using the customized Delta Frame of Reference
(DFoR) algorithm, the unused and dead sequences are compressed. The
frozen and redirected sequences cannot be compressed with DFoR because
the order of their elements is significant, and DFoR does not yet
support unsorted sequences. The theoretical compression ratio for
dfor_u16 can reach up to 16.
The new GUC wal_prune_dfor_compression controls (enables or disables)
compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least
5 when analyzing prune/freeze records in practice.
The patch series includes:
1. 0001-Implement-vect-and-uniqsortvect-containers-and-bitpack.patch
- Introduces the vect and uniqsortvect containers and the bitpack
unit used by DFoR.
2. 0002-Implement-Delta-Frame-of-Reference-compression.patch
- Implements DFoR itself.
3. 0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-prune.patch
- Applies DFoR to prune/freeze records.
Unit tests are included and integrated into the PostgreSQL build
system.
Using vect, bitpack, and DFoR for prune/freeze records is just one
example of their application. Independently of prune/freeze, these
algorithms can be effectively used by developers in other areas of
PostgreSQL.
Looking forward to discussion!
Best regards,
Evgeny Voropaev,
Tantor Labs, LLC.
Attachments:
[text/x-patch] v01-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (92.6K, 2-v01-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From aae414e83656b37013d241cea71a3a2f8961eb7b Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Sat, 7 Mar 2026 18:05:39 +0800
Subject: [PATCH v01 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/vect_templ.c | 293 +++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 +++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 48 +++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 417 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
35 files changed, 2824 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 947a2d79e29..dfd9d9a7beb 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..f86abd4857f
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ assert(width != 0);
+ assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..f024b9ebd98
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,293 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(&v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index f66c752d4a0..813ace1c917 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -460,6 +460,10 @@
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..717bfcdb4b6
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,48 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# Object files of vect, bitpack and dfor used by this unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -c $< -MMD -MP -MF $(DEPDIR)/$(*F).Po -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..a34a9905963
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,417 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args)
+ __attribute__((format(gnu_printf, 1, 0)));
+
+void tap_plan(int tests, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) __attribute__((format(gnu_printf, 4, 0)));
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 4, 5)));
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) __attribute__((format(gnu_printf, 5, 6)));
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 5, 6)));
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+int diag(const char *fmt, ...) __attribute__((format(gnu_printf, 1, 2)));
+
+int bail_out(int ignore, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+void tap_skip(int n, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+void tap_todo(int ignore, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status()
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo()
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
--
2.53.0
[text/x-patch] v01-0002-Implement-Delta-Frame-of-Reference-compression.patch (40.9K, 3-v01-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 3b01d8c20a3696dc0dca08e080e4c6785cbab59d Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Sat, 7 Mar 2026 19:10:41 +0800
Subject: [PATCH v01 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 618 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
11 files changed, 1199 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..b4bf059327d
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,618 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of Frame of Reference with Delta
+ * container and corresponding algorithm.
+ *
+ * Type of original items defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * delta is difference between the current member and the previous one. The
+ * delta for the first member (having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of undefined
+ * bits of fixed width.
+ *
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ int res;
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+
+ item_t mask;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ assert(cnt == vDeltas->cnt);
+ assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ assert(isExcUsage == DFOR_EXC_USE);
+ assert(usvExcPos.m[j] == i);
+ assert(j < usvExcPos.cnt);
+ assert(j < vExcPosDeltas.cnt);
+ assert(dfor->exc_wid != 0);
+ assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ assert(j == usvExcPos.cnt);
+ else
+ assert(j == 0);
+
+ assert(d == stats.delta_pack_nbits);
+ assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ assert(crExc == szDeltaPack + szExcPack);
+ assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 717bfcdb4b6..f7204baa207 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -27,7 +27,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v01-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (30.6K, 4-v01-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 85365c33aa0bd9703a5376fb94d1af17ea5ac3b6 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 6 Mar 2026 17:35:40 +0800
Subject: [PATCH v01 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 134 +++++++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 ++++++++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++++--
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 ++-
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 ++++++++++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 155 ++++++++++++++++++
13 files changed, 616 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index 6d39a5fff7c..bbe9d471f4e 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 65c9f393f41..7e6cd793afb 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -202,7 +203,6 @@ static void page_verify_redirects(Page page);
static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool do_hint_prune,
PruneState *prstate);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2147,6 +2147,17 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+heap_log_offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2201,11 +2212,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2230,6 +2264,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2258,26 +2296,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ qsort(dead, ndead, sizeof(OffsetNumber), heap_log_offset_cmp);
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ qsort(unused, nunused, sizeof(OffsetNumber), heap_log_offset_cmp);
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 02ae91653c1..e8e7569da9c 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 5ee84a639d8..dfd71f2924e 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3398,6 +3398,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 38aaf82f120..15fe27e11ef 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index 4c1ee649501..f6f84c989b8 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -5,8 +5,9 @@ PGAPPICON=win32
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -16,9 +17,12 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -27,6 +31,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index ce3566ba949..dd1d06860ba 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -339,6 +340,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -511,6 +514,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index f7204baa207..5e30b52d585 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -42,6 +42,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..819984aa568
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,155 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node) = @_;
+
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # Insert many tuples
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,300000) g;
+ });
+
+ # Delete 80% of records
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 550 <= 540;
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_prune_stats
+{
+ my ($node) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir ."\n";
+
+ # Find the first WAL segment
+ my @wal_files = sort glob("$wal_dir/[0-9A-F]*");
+ my $start_seg = $wal_files[0];
+
+ die "No WAL files found" unless defined $start_seg;
+
+ # Run pg_waldump on all segments
+ my $cmd = "pg_waldump --rmgr=Heap2 -p $wal_dir $start_seg 2>/dev/null";
+
+ my @lines = `$cmd`;
+
+print "lines=" . @lines . "\n";
+
+ my $records = 0;
+ my $bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ next unless $line =~ /PRUNE_VACUUM_SCAN/;
+ $records++;
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ $bytes += $1;
+ }
+ }
+ print "records=" . $records . "; bytes=" . $bytes ."\n";
+ return ($records, $bytes);
+}
+
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ });
+
+ $node->start;
+
+ generate_prune_workload($node);
+
+ $node->stop;
+
+ return collect_prune_stats($node);
+}
+
+# ------------------------------------------------------------
+# Cluster 1: compression OFF
+# ------------------------------------------------------------
+my ($off_count, $off_bytes) = run_cluster_test(
+ "prune_dfor_off",
+ "off"
+);
+
+note("Compression OFF: $off_count records, $off_bytes bytes");
+
+# ------------------------------------------------------------
+# Cluster 2: compression ON
+# ------------------------------------------------------------
+my ($on_count, $on_bytes) = run_cluster_test(
+ "prune_dfor_on",
+ "on"
+);
+
+note("Compression ON: $on_count records, $on_bytes bytes");
+
+# ------------------------------------------------------------
+# Compression ratio
+# ------------------------------------------------------------
+my $ratio = "N/A";
+
+if ($on_bytes > 0)
+{
+ $ratio = sprintf("%.2f", $off_bytes / $on_bytes);
+}
+
+note("Compression ratio (uncompressed/compressed): $ratio");
+note("Numerator (uncompressed bytes): $off_bytes");
+note("Denominator (compressed bytes): $on_bytes");
+
+# ------------------------------------------------------------
+# Expect compression benefit
+# ------------------------------------------------------------
+cmp_ok(
+ $ratio, '>=', 5,
+ 'DFOR compression should reduce the size of WAL by at least 5 times.'
+);
+
+done_testing();
\ No newline at end of file
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-03-20 10:05 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 0 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-03-20 10:05 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; pgsql-hackers
Hello Andrey,
> Great idea and nice library for DFoR! Thank you for your attention.
I wish the patch would be useful.
All your proposals and recommendations have been implemented in v02.
Also the meson settings has been updated for supporting the new
developments.
> Perhaps, first thing to start is to fix CI failures
Once meson is fixed, tests should pass successfully now. Looking
forward to this.
> As a minor nit: do not use stdlib assert(), use capital Assert()
Done.
> Are we 100% sure qsort() won't allocate something anywhere?
> sort_template.h seems to be allocation-free, but just in case...
No guarantees from libraries or from descriptions about qsort's
behaviour regarding dynamic memory allocation. So, the qsort is just
substituted with the sort_template, which we trust, and as you
proposed.
Waiting for tests to have passed, and then I hope we could move further.
Best regards, Evgeny Voropaev.
Attachments:
[text/x-patch] v02-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (95.5K, 2-v02-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From d93c4358bc706844cd3487a5c13bf890e795cf71 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v02 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 293 +++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 +++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 48 +++
src/test/dfor/meson.build | 61 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 417 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
38 files changed, 2893 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 947a2d79e29..dfd9d9a7beb 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..f9244477787
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..0929ca69e9c
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,293 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(&v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index fd6b093bb3a..d20910105fe 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -499,6 +499,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..5efc1af0a24
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,48 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND -c $< -MMD -MP -MF $(DEPDIR)/$(*F).Po -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..9802802111f
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,61 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..d0d149ec5c8
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,417 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args)
+ __attribute__((format(gnu_printf, 1, 0)));
+
+void tap_plan(int tests, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) __attribute__((format(gnu_printf, 4, 0)));
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 4, 5)));
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) __attribute__((format(gnu_printf, 5, 6)));
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 5, 6)));
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+int diag(const char *fmt, ...) __attribute__((format(gnu_printf, 1, 2)));
+
+int bail_out(int ignore, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+void tap_skip(int n, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+void tap_todo(int ignore, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 2, 3)));
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(gnu_printf, 6, 7)));
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v02-0002-Implement-Delta-Frame-of-Reference-compression.patch (41.9K, 3-v02-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From f50e2e11618acf8fbe1bff4216872bb85600474d Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v02 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 617 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
13 files changed, 1201 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..21a30a224c5
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,617 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of Frame of Reference with Delta
+ * container and corresponding algorithm.
+ *
+ * Type of original items defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * delta is difference between the current member and the previous one. The
+ * delta for the first member (having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of undefined
+ * bits of fixed width.
+ *
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 5efc1af0a24..f9f1b705c5b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -27,7 +27,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index 9802802111f..3ff36c439c8 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v02-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (32.7K, 4-v02-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From e58175fb7c93678f6f982a7977206c92876e90cd Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v02 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 141 ++++++++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 ++++++++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++++--
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 ++-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 ++++++++++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 155 ++++++++++++++++++
16 files changed, 627 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index 1da774c1536..7f96a30dd71 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 8d9f0694206..c17573d20ec 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -195,7 +196,6 @@ static void page_verify_redirects(Page page);
static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool do_hint_prune,
PruneState *prstate);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2130,6 +2130,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2184,11 +2202,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2213,6 +2254,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2241,26 +2286,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 02ae91653c1..e8e7569da9c 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 0c9854ad8fc..68b6713b536 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3413,6 +3413,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 1e14b7b4af0..86a509fb9f7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e4abe6c0077..2034ccd4933 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -255,6 +255,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index 4c1ee649501..f6f84c989b8 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -5,8 +5,9 @@ PGAPPICON=win32
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -16,9 +17,12 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -27,6 +31,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 633a9874bb5..0f34ccd6c4f 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -9,6 +9,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index ce3566ba949..dd1d06860ba 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -339,6 +340,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -511,6 +514,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index f9f1b705c5b..705f936ec11 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -42,6 +42,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..819984aa568
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,155 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node) = @_;
+
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # Insert many tuples
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,300000) g;
+ });
+
+ # Delete 80% of records
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 550 <= 540;
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_prune_stats
+{
+ my ($node) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir ."\n";
+
+ # Find the first WAL segment
+ my @wal_files = sort glob("$wal_dir/[0-9A-F]*");
+ my $start_seg = $wal_files[0];
+
+ die "No WAL files found" unless defined $start_seg;
+
+ # Run pg_waldump on all segments
+ my $cmd = "pg_waldump --rmgr=Heap2 -p $wal_dir $start_seg 2>/dev/null";
+
+ my @lines = `$cmd`;
+
+print "lines=" . @lines . "\n";
+
+ my $records = 0;
+ my $bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ next unless $line =~ /PRUNE_VACUUM_SCAN/;
+ $records++;
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ $bytes += $1;
+ }
+ }
+ print "records=" . $records . "; bytes=" . $bytes ."\n";
+ return ($records, $bytes);
+}
+
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ });
+
+ $node->start;
+
+ generate_prune_workload($node);
+
+ $node->stop;
+
+ return collect_prune_stats($node);
+}
+
+# ------------------------------------------------------------
+# Cluster 1: compression OFF
+# ------------------------------------------------------------
+my ($off_count, $off_bytes) = run_cluster_test(
+ "prune_dfor_off",
+ "off"
+);
+
+note("Compression OFF: $off_count records, $off_bytes bytes");
+
+# ------------------------------------------------------------
+# Cluster 2: compression ON
+# ------------------------------------------------------------
+my ($on_count, $on_bytes) = run_cluster_test(
+ "prune_dfor_on",
+ "on"
+);
+
+note("Compression ON: $on_count records, $on_bytes bytes");
+
+# ------------------------------------------------------------
+# Compression ratio
+# ------------------------------------------------------------
+my $ratio = "N/A";
+
+if ($on_bytes > 0)
+{
+ $ratio = sprintf("%.2f", $off_bytes / $on_bytes);
+}
+
+note("Compression ratio (uncompressed/compressed): $ratio");
+note("Numerator (uncompressed bytes): $off_bytes");
+note("Denominator (compressed bytes): $on_bytes");
+
+# ------------------------------------------------------------
+# Expect compression benefit
+# ------------------------------------------------------------
+cmp_ok(
+ $ratio, '>=', 5,
+ 'DFOR compression should reduce the size of WAL by at least 5 times.'
+);
+
+done_testing();
\ No newline at end of file
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-03-22 16:20 Andres Freund <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Andres Freund @ 2026-03-22 16:20 UTC (permalink / raw)
To: Evgeny Voropaev <[email protected]>; +Cc: pgsql-hackers
Hi,
On 2026-03-07 21:56:18 +0800, Evgeny Voropaev wrote:
> A prune/freeze record contains four sequences of integers representing
> frozen, redirected, unused, and dead tuples. Currently, these
> sequences are uncompressed, which adds overhead. Eliminating empty
> leading bits in these offsets can reduce the size of the record, and
> applying bit-packing algorithms can reduce it even further. Reducing
> WAL record size also lessens the load on disk and network, which are
> heavily used for storing WAL and transmitting it to replicas. For
> example, with BLCKSZ equal to 8192, the maximum tuple offset is around
> 580. Despite this, all offsets are stored as 16-bit integers with no
> compression applied.
I'm unconvinced that this is a serious problem - typically the overhead of WAL
volume due to pruning / freezing is due to the full page images emitted, not
the raw size of the records. Once an FPI is emitted, this doesn't matter.
What gains have you measured in somewhat realistic workloads?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-03-24 14:28 Evgeny Voropaev <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 3 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-03-24 14:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
Hello Andres,
> I'm unconvinced that this is a serious problem - typically the overhead of WAL
> volume due to pruning / freezing is due to the full page images emitted, not
> the raw size of the records. Once an FPI is emitted, this doesn't matter.
>
> What gains have you measured in somewhat realistic workloads?
So far, we have had no tests in any real production environment.
Moreover, the load in the new test
(recovery/t/052_prune_dfor_compression.pl) is quite contrived. However,
it demonstrates a compression ratio of more than 5, and it is measured
for an overall size of all prune/freeze records with no filtering.
Further development is the implementation of compression of unsorted
sequences. This is going to allow PostgreSQL to compress also the
'frozen' and the 'redirected' offset sequences, which should result in a
greater compression ratio.
But I agree with you, Andres, we need practical results to estimate a
profit. I wish we would test it on some real load soon.
Also I hope, independently of its usage in prune/freeze records, the
DFoR itself might be used for compression sequences in other places of PG.
Best regards,
Evgeny Voropaev.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-03-24 16:25 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
2 siblings, 0 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-03-24 16:25 UTC (permalink / raw)
To: pgsql-hackers; Andrey Borodin <[email protected]>; Andres Freund <[email protected]>
Rebase onto 2102ebb1953.
Fix warnings about gnu_printf in tap.c.
Fix wrong destination pointer in memcpy in vect_init.
Fix intl library linking problem for the dfor tests under FreBSD.
Attachments:
[text/x-patch] v04-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (95.6K, 2-v04-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From 38f141a8367f8b4eba7d5c873261e446e04dea9d Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v04 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 ++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 +++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 48 +++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 417 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
38 files changed, 2902 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index a7699b026bb..a37142f8160 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..f9244477787
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index fd6b093bb3a..d20910105fe 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -499,6 +499,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..5efc1af0a24
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,48 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND -c $< -MMD -MP -MF $(DEPDIR)/$(*F).Po -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..a608929a2df
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,417 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args)
+ __attribute__((format(printf, 1, 0)));
+
+void tap_plan(int tests, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) __attribute__((format(printf, 4, 0)));
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ __attribute__((format(printf, 4, 5)));
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) __attribute__((format(printf, 5, 6)));
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(printf, 5, 6)));
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ __attribute__((format(printf, 6, 7)));
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ __attribute__((format(printf, 6, 7)));
+
+int diag(const char *fmt, ...) __attribute__((format(printf, 1, 2)));
+
+int bail_out(int ignore, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+void tap_skip(int n, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+void tap_todo(int ignore, const char *fmt, ...)
+ __attribute__((format(printf, 2, 3)));
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ __attribute__((format(printf, 6, 7)));
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v04-0002-Implement-Delta-Frame-of-Reference-compression.patch (41.9K, 3-v04-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From b8b5e3231cffc96a05d41212f85889b3c05df146 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v04 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 617 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
13 files changed, 1201 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..21a30a224c5
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,617 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of Frame of Reference with Delta
+ * container and corresponding algorithm.
+ *
+ * Type of original items defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * delta is difference between the current member and the previous one. The
+ * delta for the first member (having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of undefined
+ * bits of fixed width.
+ *
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 5efc1af0a24..f9f1b705c5b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -27,7 +27,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v04-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (32.8K, 4-v04-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 6969eb48d91982ff6c800d1c80e065cb94932fcc Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v04 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 141 ++++++++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 ++++++++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++++--
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 ++-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 ++++++++++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 155 ++++++++++++++++++
16 files changed, 627 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index 1da774c1536..7f96a30dd71 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index b383b0fca8b..42b871c4283 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -232,7 +233,6 @@ static void page_verify_redirects(Page page);
static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool do_hint_prune,
PruneState *prstate);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2406,6 +2406,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2460,11 +2478,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2489,6 +2530,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2517,26 +2562,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 02ae91653c1..e8e7569da9c 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 0c9854ad8fc..68b6713b536 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3413,6 +3413,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 1e14b7b4af0..86a509fb9f7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e4abe6c0077..2034ccd4933 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -255,6 +255,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index ce3566ba949..dd1d06860ba 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -339,6 +340,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -511,6 +514,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index f9f1b705c5b..705f936ec11 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -42,6 +42,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..819984aa568
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,155 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node) = @_;
+
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # Insert many tuples
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,300000) g;
+ });
+
+ # Delete 80% of records
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 550 <= 540;
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_prune_stats
+{
+ my ($node) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir ."\n";
+
+ # Find the first WAL segment
+ my @wal_files = sort glob("$wal_dir/[0-9A-F]*");
+ my $start_seg = $wal_files[0];
+
+ die "No WAL files found" unless defined $start_seg;
+
+ # Run pg_waldump on all segments
+ my $cmd = "pg_waldump --rmgr=Heap2 -p $wal_dir $start_seg 2>/dev/null";
+
+ my @lines = `$cmd`;
+
+print "lines=" . @lines . "\n";
+
+ my $records = 0;
+ my $bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ next unless $line =~ /PRUNE_VACUUM_SCAN/;
+ $records++;
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ $bytes += $1;
+ }
+ }
+ print "records=" . $records . "; bytes=" . $bytes ."\n";
+ return ($records, $bytes);
+}
+
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ });
+
+ $node->start;
+
+ generate_prune_workload($node);
+
+ $node->stop;
+
+ return collect_prune_stats($node);
+}
+
+# ------------------------------------------------------------
+# Cluster 1: compression OFF
+# ------------------------------------------------------------
+my ($off_count, $off_bytes) = run_cluster_test(
+ "prune_dfor_off",
+ "off"
+);
+
+note("Compression OFF: $off_count records, $off_bytes bytes");
+
+# ------------------------------------------------------------
+# Cluster 2: compression ON
+# ------------------------------------------------------------
+my ($on_count, $on_bytes) = run_cluster_test(
+ "prune_dfor_on",
+ "on"
+);
+
+note("Compression ON: $on_count records, $on_bytes bytes");
+
+# ------------------------------------------------------------
+# Compression ratio
+# ------------------------------------------------------------
+my $ratio = "N/A";
+
+if ($on_bytes > 0)
+{
+ $ratio = sprintf("%.2f", $off_bytes / $on_bytes);
+}
+
+note("Compression ratio (uncompressed/compressed): $ratio");
+note("Numerator (uncompressed bytes): $off_bytes");
+note("Denominator (compressed bytes): $on_bytes");
+
+# ------------------------------------------------------------
+# Expect compression benefit
+# ------------------------------------------------------------
+cmp_ok(
+ $ratio, '>=', 5,
+ 'DFOR compression should reduce the size of WAL by at least 5 times.'
+);
+
+done_testing();
\ No newline at end of file
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-08 12:34 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
2 siblings, 2 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-08 12:34 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
Tomas, Andreus, Andrey, hello!
> A ~170kB patch really should present some numbers
> quantifying the expected benefit. It doesn't need to be a real workload
> from production, but something plausible enough. Even some basic
> back-of-the-envelope calculations might be enough to show the promise.
The patch results in reduction of WAL total size by:
81% during vacuuming a table having no index,
and by 55% during vacuuming a table having an index.
The numbers are the next:
=== VACUUM (table with no index) ===
-------------------- ----------------- ----------------- -----------
DFOR off, bytes DFOR on, bytes Reduction
-------------------- ----------------- ----------------- -----------
WAL total size 6743149 1184446 82%
Prune records size 6710185 1159723 5.8x
-------------------- ----------------- ----------------- -----------
=== VACUUM (table with index) ===
-------------------- ----------------- ----------------- -----------
DFOR off, bytes DFOR on, bytes Reduction
-------------------- ----------------- ----------------- -----------
WAL total size 20394208 8907090 56%
Prune records size 6812850 1225944 5.6x
-------------------- ----------------- ----------------- -----------
The logic of the tests is based on the technique from [1] and is the
next:
-- SQL
CREATE TABLE t_prune ( id int, val text )
WITH (fillfactor = 100, autovacuum_enabled = false);
INSERT INTO t_prune
SELECT g, 'x' FROM generate_series(1,3000000) g;
CREATE INDEX ON t_prune(id); -- for the test using an indexed table
DELETE FROM t_prune WHERE id % 500 <> 0;
SELECT pg_current_wal_flush_lsn(); -- get start_lsn here
VACUUM FREEZE t_prune; -- 3 times
SELECT pg_current_wal_flush_lsn(); -- get end_lsn here
# BASH
# stop cluster
pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null;
The test is implemented in 052_prune_dfor_compression.pl, therefore
the presented results can be refetched by restarting this test script.
> Also, I find it somewhat unlikely we'd import a GPLv3 library like
> this, even if it's just a testing framework. Even ignoring the
> question of having a different license for some of the code, it'd mean
> maintenance burden (maybe libtap is stable/mature, no idea). I don't
> see why this would be better than "write a SQL callable test module".
I am ready to rework it once there is consensus on the core of the
patch.
Best regards,
Evgeny.
P.s. rebased onto a1643d40b30.
[1]
https://www.postgresql.org/message-id/flat/CAAKRu_ZMw6Npd_qm2KM%2BFwQ3cMOMx1Dh3VMhp8-V7SOLxdK9-g%40m...
Attachments:
[text/x-patch] v08-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (95.6K, 2-v08-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From 2a26973165dc0a84d27a78412495388a5eb01436 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v08 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 ++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 ++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 53 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 +++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
38 files changed, 2911 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index a7699b026bb..a37142f8160 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..5f721ea1ebc
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 88d13ec9993..1e48a52ae90 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..3b2c035927f
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,53 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+DEPDIR ?= .deps # This fixes a problem in some CI jobs
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $< \
+ -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v08-0002-Implement-Delta-Frame-of-Reference-compression.patch (42.1K, 3-v08-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 135c6e7bfdec3d71774a28ae81de038b7a42a79a Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v08 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 618 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
13 files changed, 1202 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..dae79adfb66
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,618 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The type of original items is defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * The delta is a difference between the current item and the previous one.
+ * The delta of the first item (the member having the zero index) is its actual
+ * value: delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of
+ * bits. Each serialised delta in 'deltas' has a fixed bit width. If the delta's
+ * width exceeds the allowed size of a delta in 'deltas', the higher bits of this
+ * delta is put into exceptions.
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 3b2c035927f..43f0d51fb70 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -32,7 +32,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v08-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (35.6K, 4-v08-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 6d16b64e2fad8018999c604731ed3cc239ab89e1 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v08 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 141 ++++++++-
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++-
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 755 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index f3f419d3dc1..2ecbd277c02 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..a7de9ea7b05 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2663,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2650,26 +2695,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..2446f158720 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..5ce885e6324 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -494,6 +497,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 43f0d51fb70..ee88f8b77cd 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -47,6 +47,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-09 04:21 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-09 04:21 UTC (permalink / raw)
To: Andres Freund <[email protected]>; pgsql-hackers
Hello hackers!
Dear Andres, please forgive me for the typo in your name in my previous
message. It was not intentional. My blushes!
v09 is the same as v08, except for a difference in test/dfor/Makefile.
I am still trying to fix the build errors in CI/CD.
P.s. rebased onto e0fa5bd1465.
Attachments:
[text/x-patch] v09-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (95.9K, 2-v09-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From e116727dfc0fd49ac51c9bb5a5d3a8c5563c0ae2 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v09 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 ++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 ++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 61 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 +++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
38 files changed, 2919 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index a7699b026bb..a37142f8160 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..5f721ea1ebc
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 88d13ec9993..1e48a52ae90 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..a15d5eaf736
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,61 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+DEPDIR ?= .deps # This fixes a problem in some CI jobs
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+# $(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+# @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+# $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+# -c $< \
+# -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+# -o $@
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ echo "The rule *.c->*.o is working for $@. The DEP's calculation resulted in $(DEPDIR)/$(@F:.o=.Po)"
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $(dfor_dir)/$(@F:.o=.c) \
+ -MMD -MP -MF $(DEPDIR)/$(@F:.o=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v09-0002-Implement-Delta-Frame-of-Reference-compression.patch (42.1K, 3-v09-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 92ae9aab046c3ece46cc798426c040947adccb2b Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v09 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 618 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
13 files changed, 1202 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..dae79adfb66
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,618 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The type of original items is defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * The delta is a difference between the current item and the previous one.
+ * The delta of the first item (the member having the zero index) is its actual
+ * value: delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of
+ * bits. Each serialised delta in 'deltas' has a fixed bit width. If the delta's
+ * width exceeds the allowed size of a delta in 'deltas', the higher bits of this
+ * delta is put into exceptions.
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index a15d5eaf736..1af529c7378 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -40,7 +40,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v09-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (35.6K, 4-v09-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From db62f0d6be38f1e01a18f8ce24e5b03b5cb93595 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v09 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 141 ++++++++-
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++-
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 755 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index f3f419d3dc1..2ecbd277c02 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..a7de9ea7b05 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2663,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2650,26 +2695,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..2446f158720 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..5ce885e6324 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -494,6 +497,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 1af529c7378..cec83d43319 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -55,6 +55,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-09 10:00 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
0 siblings, 0 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-09 10:00 UTC (permalink / raw)
To: pgsql-hackers
The problem with GCC dependencies appearing in Debian Trixie in
test/dfor has been replayed locally and fixed.
P.S. Rebased onto 11d6042337f.
Attachments:
[text/x-patch] v10-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (95.6K, 2-v10-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From cf50336d2eede0316adf7b5cdda2a2ae65dbd934 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v10 1/3] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allow developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Unit tests for the provided implementations are included. Unit tests are
implemented as binary applications written in C language
(ELF executables) that support the TAP protocol and are run using the
Prove utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 ++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 ++++
src/include/lib/bitpack_templ.h | 14 +
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 ++
src/include/lib/vect_templ_staple.h | 140 ++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 ++
src/include/lib/vect_u16_config.h | 10 +
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 54 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 +++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 +++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 +++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
38 files changed, 2912 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index a7699b026bb..a37142f8160 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..5f721ea1ebc
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 88d13ec9993..1e48a52ae90 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..4fc9f4bc1ba
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,54 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# This fixes a problem in some CI jobs
+DEPDIR ?= ".deps"
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $< \
+ -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v10-0002-Implement-Delta-Frame-of-Reference-compression.patch (42.1K, 3-v10-0002-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From f4ecbd097f9ff5a2ec4d74f7d6b026a69b2a7674 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v10 2/3] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 618 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 +++++++++++++++++
13 files changed, 1202 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..dae79adfb66
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,618 @@
+/*
+ * dfor.c
+ *
+ * DFOR_TEMPL implements the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The type of original items is defined with the item_t macro. item_t must be
+ * an unsigned integer (uint8_t, uint16_t, ... uint64_t)
+ *
+ * Each bit vector, having been serialised, represents next structure:
+ *
+ * | deltas | exceptions | exceptions positions |
+ *
+ * The delta is a difference between the current item and the previous one.
+ * The delta of the first item (the member having the zero index) is its actual
+ * value: delta[0] = m[0]-0 = m[0]. A having serialised delta is a sequence of
+ * bits. Each serialised delta in 'deltas' has a fixed bit width. If the delta's
+ * width exceeds the allowed size of a delta in 'deltas', the higher bits of this
+ * delta is put into exceptions.
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 4fc9f4bc1ba..2ca98f76a0f 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -33,7 +33,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v10-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (35.6K, 4-v10-0003-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 87af871c4a4c1b6f7a8b4285925064d038a654ea Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v10 3/3] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 12 +-
src/backend/access/heap/pruneheap.c | 141 ++++++++-
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 49 ++-
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 8 +-
src/include/access/heapam_xlog_dfor.h | 137 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 755 insertions(+), 34 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index f3f419d3dc1..2ecbd277c02 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,11 +106,20 @@ heap_xlog_prune_freeze(XLogReaderState *record)
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ /*
+ * DFoR unpacking needs outer buffers for saving results and for
+ * allocating containers used during decompression. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three chunks are needed for internal needs of the
+ * dfor_unpack function.
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused, dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..a7de9ea7b05 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,34 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2663,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2650,26 +2695,92 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE, dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
+
if (nfrozen > 0)
XLogRegisterBufData(0, frz_offsets,
sizeof(OffsetNumber) * nfrozen);
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..2446f158720 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -108,7 +109,8 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
@@ -146,14 +148,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,14 +173,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
@@ -309,13 +327,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..5ce885e6324 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -494,6 +497,7 @@ extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 2ca98f76a0f..f7a287a1d3b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -48,6 +48,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-09 16:50 Andres Freund <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 1 reply; 17+ messages in thread
From: Andres Freund @ 2026-04-09 16:50 UTC (permalink / raw)
To: Evgeny Voropaev <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
On 2026-04-08 20:34:42 +0800, Evgeny Voropaev wrote:
> Tomas, Andreus, Andrey, hello!
>
> > A ~170kB patch really should present some numbers
> > quantifying the expected benefit. It doesn't need to be a real workload
> > from production, but something plausible enough. Even some basic
> > back-of-the-envelope calculations might be enough to show the promise.
>
> The patch results in reduction of WAL total size by:
> 81% during vacuuming a table having no index,
> and by 55% during vacuuming a table having an index.
>
> The numbers are the next:
>
> === VACUUM (table with no index) ===
> -------------------- ----------------- ----------------- -----------
> DFOR off, bytes DFOR on, bytes Reduction
> -------------------- ----------------- ----------------- -----------
> WAL total size 6743149 1184446 82%
> Prune records size 6710185 1159723 5.8x
> -------------------- ----------------- ----------------- -----------
>
> === VACUUM (table with index) ===
> -------------------- ----------------- ----------------- -----------
> DFOR off, bytes DFOR on, bytes Reduction
> -------------------- ----------------- ----------------- -----------
> WAL total size 20394208 8907090 56%
> Prune records size 6812850 1225944 5.6x
> -------------------- ----------------- ----------------- -----------
These numbers make the impact sound bigger than I think it really is:
- They neglect that the insert generates ~183MB of WAL, the delete ~161MB
without indexes and ~243MB / 161MB with. In contrast to that 6.7Mb isn't
particularly significant.
- Workloads deleting almost all records in the table but leaving some in to
prevent truncation aren't particularly common.
- The narrowness of the rows (~30 bytes, with row header) makes the wins much
bigger than they'd be in realistic cases
- The workload doesn't involve any FPIs. It's much more common to have
vacuum's occur later and trigger FPIs.
Heh. In this case FPIs actually would *reduce* the overhead of the current
code, because the page is so empty after all the deletes that the FPI uses
less space than the update . It's 4.1MB when not using indexes and not using
wal compression and 1MB with wal compression.
Seems we could get a fair bit of benefit by just using a heuristic to switch
to an FPI when there are enough changes.
I think that'd just be a few lines.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-09 17:55 Andrey Borodin <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Andrey Borodin @ 2026-04-09 17:55 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Evgeny Voropaev <[email protected]>; Tomas Vondra <[email protected]>; pgsql-hackers
> On 9 Apr 2026, at 21:50, Andres Freund <[email protected]> wrote:
>
> These numbers make the impact sound bigger than I think it really is:
>
> - They neglect that the insert generates ~183MB of WAL, the delete ~161MB
> without indexes and ~243MB / 161MB with. In contrast to that 6.7Mb isn't
> particularly significant.
Well, vacuuming of a bloated tables does happen. The amount of WAL needed to
accumulate bloat does not invalidate benefits of reducing WAL needed to vacuum
bloat.
> - Workloads deleting almost all records in the table but leaving some in to
> prevent truncation aren't particularly common.
Queue tables may be kind of antipattern, yet users use such tables. And sometimes
they tend to have ~90% bloat. And cause 99% of problems.
> - The narrowness of the rows (~30 bytes, with row header) makes the wins much
> bigger than they'd be in realistic cases
It’s crafted benchmark to demonstrate bright side. It’s also super easy to demonstrate
the case when proposed patch does not give any benefit at all.
Evgeny, do you know of any cases when the patch has negative effect?
I think if it’s strictly non-negative - then we can just weight complexity of maintaining
the proposed approach against benefits.
> - The workload doesn't involve any FPIs. It's much more common to have
> vacuum's occur later and trigger FPIs.
AFAIU, without special handling FPIs will not substitute xl_heap_prune, will they?
> Heh. In this case FPIs actually would *reduce* the overhead of the current
> code, because the page is so empty after all the deletes that the FPI uses
> less space than the update . It's 4.1MB when not using indexes and not using
> wal compression and 1MB with wal compression.
>
> Seems we could get a fair bit of benefit by just using a heuristic to switch
> to an FPI when there are enough changes.
I think we could even do it in a generic way: if the record body is heavier that its FPIs - just
do FPIs only.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-10 08:44 Evgeny Voropaev <[email protected]>
parent: Andrey Borodin <[email protected]>
0 siblings, 1 reply; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-10 08:44 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; Andres Freund <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers
> Evgeny, do you know of any cases when the patch has negative effect?
>
> I think if it’s strictly non-negative - then we can just weight complexity of maintaining
> the proposed approach against benefits.
Andrey, during my research and testing I haven't encountered any
negative effects from this patch on the WAL size.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-14 07:08 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
0 siblings, 0 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-14 07:08 UTC (permalink / raw)
To: pgsql-hackers
Continue fixing CI issues in this patch. Resolved frz_offsets
misalignment caused by odd byte counts in dead or unused tuple packs.
Patch set has been split, unit-tests have been moved into separated
patch-files. It shows that the very essence of the patch has size of
90 KB only.
P.S. Rebased onto fce3f7d2677.
Attachments:
[text/x-patch] v11-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (26.9K, 2-v11-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From 3bfa164e69fd5a7dfad6a1dd31b2ffb9d6f5575f Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:12:55 +0800
Subject: [PATCH v11 1/5] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allows developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 +++++++++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_staple_templ.h | 57 +++++
src/include/lib/bitpack_templ.h | 14 ++
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 +++
src/include/lib/vect_templ_staple.h | 140 ++++++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 34 +++
src/include/lib/vect_u16_config.h | 10 +
18 files changed, 824 insertions(+)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_staple_templ.h
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..5f721ea1ebc
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_staple_templ.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 88d13ec9993..1e48a52ae90 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_staple_templ.h b/src/include/lib/bitpack_staple_templ.h
new file mode 100644
index 00000000000..5c9972e08cb
--- /dev/null
+++ b/src/include/lib/bitpack_staple_templ.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_staple_templ.h.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..b3a6e06c328
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_staple_templ.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..edf81a417f4
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,34 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+/* clang-format off */
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+/* clang-format on */
+
+/*
+ * Types are supposed to be created created by this file
+ * vect_u16_t
+ * item_u16_t
+ * uniqsortvect_u16_t
+ */
+/*
+ * Functions are supposed to be created by this file
+ * vect_u16_create
+ * vect_u16_create_filled
+ * vect_u16_reserve
+ * vect_u16_append
+ * vect_u16_destroy
+ * vect_u16_print
+ * vect_u16_compare
+ * vect_u16_insert
+ * vect_u16_clear
+ * usv_u16_insert
+ * usv_u16_search
+ */
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
--
2.53.0
[text/x-patch] v11-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch (69.2K, 3-v11-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch)
download | inline diff:
From 7c1e2f16fc37302cd2816dc8f8142ff6c38583f9 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:18:42 +0800
Subject: [PATCH v11 2/5] Tests of vect and uniqsortvect containers and of
bitpack algorithms.
Unit tests for the vect and the bitpack units are imlemented. Unit tests
are implemented as binary applications written in C language (ELF
executables) that support the TAP protocol and are run using the Prove
utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 54 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 ++++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 ++++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
20 files changed, 2088 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index cef1ad7f87d..0b6a22be18b 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..4fc9f4bc1ba
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,54 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# This fixes a problem in some CI jobs
+DEPDIR ?= ".deps"
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $< \
+ -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v11-0003-Implement-Delta-Frame-of-Reference-compression.patch (28.7K, 4-v11-0003-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 683e96c12cbb5dcf8d9baaa5e5d46eeab66781d9 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:15:34 +0800
Subject: [PATCH v11 3/5] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
The current implementation of the DFOR algorithm supports items of
unsigned integer types. The type of an item is defined with the
item_t macro.
The compressed pack comprises three parts:
- deltas;
- exceptions positions.
- exceptions;
Each of them is a densily bit-packed sequence.
The delta is a difference between the current item and the previous one. The
delta of the first item (the item having the zero index) is its actual value:
delta[0] = m[0]-0 = m[0]. Serialised deltas is a sequence of bits. Each
serialised delta in 'deltas' has a fixed bit width. If the delta's width
exceeds the allowed size of a delta in 'deltas', the higher bits of this
delta is put into exceptions, its positions is saved into the 'excepton
positions' section. Usage of exceptions can be deliberately turned off.
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 633 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 125 ++++++
src/include/lib/dfor_templ_undef.h | 29 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
9 files changed, 841 insertions(+)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..1cbacbc3985
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,633 @@
+/*
+ * dfor_templ.c
+ *
+ * Implement the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The current implementation of DFOR algorithm supports items of unsigned
+ * integer type. The type of original items is defined with the item_t macro.
+ *
+ * The compressed pack comprises three parts:
+ * - deltas;
+ * - exceptions positions.
+ * - exceptions;
+ * Each of them is a densily bit-packed sequence.
+ *
+ * The delta is a difference between the current item and the previous one. The
+ * delta of the first item (the item having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. Serialised deltas is a sequence of bits. Each
+ * serialised delta in 'deltas' has a fixed bit width. If the delta's width
+ * exceeds the allowed size of a delta in 'deltas', the higher bits of this
+ * delta is put into exceptions, its positions is saved into the 'excepton
+ * positions' section. Usage of exceptions can be deliberately turned off.
+ *
+ * DFoR supports both external memory (outer memory) provided by a caller
+ * and automatically managed memory, allocated by means of malloc, palloc
+ * or similar functions. Memory management configuration must be defined
+ * during initialization. All subsequent operations follow this
+ * configuration. For example, a caller can place a buffer on the stack to
+ * avoid heap allocation and pass the buffer to a DFoR unit. As a result,
+ * the packing and unpacking processes exclude dynamic allocation.
+ *
+ * The DFoR unit is implemented as a set of templates. Developers can
+ * generate DFoR implementations for any unsigned integer type (uint8_t,
+ * uint16_t, uint32_t, uint64_t).
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Don't calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL)
+ return -1;
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++) {
+ delta = arr[j] - prev;
+ vect_append(vDeltas, delta);
+ prev = arr[j];
+ width = width_from_val(delta);
+
+ if (vWidthCounters == NULL) {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ } else {
+ insWidthInsert = usv_insert(usvDeltaWidths, width_from_val(delta));
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t mDelta;
+ item_t mSum = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ mDelta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ mDelta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+ mSum += mDelta;
+ vect_append(vVals, mSum);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..b4c1d41c1d3
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e93c40ac034
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,125 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..d60d3619308
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,29 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_create
+#undef vect_create_filled
+#undef vect_reserve
+#undef vect_append
+#undef vect_destroy
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
--
2.53.0
[text/x-patch] v11-0004-Tests-for-Delta-Frame-of-Reference-unit.patch (15.4K, 5-v11-0004-Tests-for-Delta-Frame-of-Reference-unit.patch)
download | inline diff:
From 523c5a08010e8882e9bf2f601f1e85092a5c61b3 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:18:07 +0800
Subject: [PATCH v11 4/5] Tests for Delta Frame of Reference unit.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 371 ++++++++++++++++++++++++++++++++++
4 files changed, 376 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 4fc9f4bc1ba..2ca98f76a0f 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -33,7 +33,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..322b714ba38
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,371 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[]);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[])
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(130);
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION\n");
+ {
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2,
+ 2 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 } /* marWidthsStatExpected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL /* marWidthsStatExpected */);
+ }
+ printf("Test DELTA CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+ {
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, 3, (uint16_t[]) { 1, 2, 7 });
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, 3, (uint16_t[]) { 4, 6, 4 }); // 4+6+4=14
+
+ res = dfor_u16_calc_width(14, &usvDeltaWidths, &vWidthCounters, &width,
+ &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", 7, "Widths={1,2,7}, Counters={4,6,4} => width=7");
+ cmp_ok(cntExceptions, "==", 0,
+ "Widths={1,2,7}, Counters={4,6,4} => excptions_num=0");
+ }
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v11-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (37.8K, 6-v11-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 63e34c398006a281dafffb6fe580a6adfc7279c4 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v11 5/5] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 21 +-
src/backend/access/heap/pruneheap.c | 153 ++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 109 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 63 ++--
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 10 +-
src/include/access/heapam_xlog_dfor.h | 137 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 782 insertions(+), 44 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index f3f419d3dc1..f99b3420cad 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -104,12 +105,26 @@ heap_xlog_prune_freeze(XLogReaderState *record)
OffsetNumber *frz_offsets;
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ char *cursor PG_USED_FOR_ASSERTS_ONLY;
- heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
+ /*
+ * Provide DFoR unpacking with outer buffers. 2 buffer parts are
+ * intended for saving sequences of offsets of dead and unused tuples.
+ * Additional three parts are for internal needs of the dfor_unpack
+ * function.
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
+ cursor = heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf_aligned.dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
@@ -155,7 +170,7 @@ heap_xlog_prune_freeze(XLogReaderState *record)
}
/* There should be no more data */
- Assert((char *) frz_offsets == dataptr + datalen);
+ Assert(cursor == dataptr + datalen);
/*
* The critical integrity requirement here is that we must never end
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..a088c3516da 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,38 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t dead_meta = { 0 };
+ dfor_meta_t unused_meta = { 0 };
+
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 2 * DFOR_BUF_PART_SIZE
+ * - for 2 packed sequences: dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 5 * DFOR_BUF_PART_SIZE
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression && (ndead > 9 || nunused > 9))
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2667,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2639,6 +2688,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
offsetof(xlhp_freeze_plans, plans));
XLogRegisterBufData(0, plans,
sizeof(xlhp_freeze_plan) * nplans);
+
+ XLogRegisterBufData(0, frz_offsets,
+ sizeof(OffsetNumber) * nfrozen);
+
}
if (nredirected > 0)
{
@@ -2650,29 +2703,93 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, redirected,
sizeof(OffsetNumber[2]) * nredirected);
}
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ /*
+ * Dead tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * dead_meta.pack = dfor_buf + DFOR_BUF_PART_SIZE;
+ */
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf);
+ }
+
+ /*
+ * Unused tuple offsets are subject to be packed with DFoR.
+ * After that we have:
+ * unused_meta.pack = dfor_buf + 2 * DFOR_BUF_PART_SIZE;
+ */
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ DFOR_BUF_PART_SIZE);
+ }
+
+ if (dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into log */
+ size_t meta_pack_sz;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
- if (nfrozen > 0)
- XLogRegisterBufData(0, frz_offsets,
- sizeof(OffsetNumber) * nfrozen);
/*
* Prepare the main xl_heap_prune record. We already set the XLHP_HAS_*
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..47fa000e367
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,109 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..33e97bb4546 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -102,17 +103,19 @@ plan_elem_desc(StringInfo buf, void *plan, void *data)
* This is in heapdesc.c so it can be shared between heap2_redo and heap2_desc
* code, the latter of which is used in frontend (pg_waldump) code.
*/
-void
+char *
heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
- xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *) cursor;
+ int nfrozen = 0;
+ xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *)cursor;
*nplans = freeze_plans->nplans;
Assert(*nplans > 0);
@@ -120,6 +123,12 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
cursor += offsetof(xlhp_freeze_plans, plans);
cursor += sizeof(xlhp_freeze_plan) * *nplans;
+
+ for (int i = 0; i < *nplans; i++)
+ nfrozen += (*plans)[i].ntuples;
+
+ *frz_offsets = (OffsetNumber *) cursor;
+ cursor += sizeof(OffsetNumber) * nfrozen;
}
else
{
@@ -146,14 +155,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf);
+ }
}
else
{
@@ -163,22 +180,29 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(!(flags & XLHP_DFOR_COMPRESSED))
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
+ else
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ }
}
else
{
*nunused = 0;
*nowunused = NULL;
}
-
- *frz_offsets = (OffsetNumber *) cursor;
+ return cursor;
}
void
@@ -309,13 +333,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..2233ff7d108 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -489,11 +492,12 @@ extern const char *heap2_identify(uint8 info);
extern void heap_xlog_logical_rewrite(XLogReaderState *r);
/* in heapdesc.c, so it can be shared between frontend/backend code */
-extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
+extern char * heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..274b14e891e
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,137 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage. Since we
+ can't calculate
+ * MaxHeapTuplesPerPage on preprocessor stage, we intentionally overestimate it
+ * as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count = 1366, and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount).
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|-------------------|--------|-----------------|
+ * | main | 0, 1 | 0-10 | item count | 11 bit | 1...1365 |
+ * | main | 1 | 11-14 | delta width | 4 bit | 1...11 |
+ * | main | 1 | 15 | extra sect. flag | 1 bit | 0...1 |
+ * | extra | 2 | 16-23 | exception count | 8 bit | 0...137 |
+ * | extra | 3 | 24-27 | exception width | 4 bit | 0...10 |
+ * | extra | 3, 4 | 28-35 | except pos. width | 4 bit | 1...11 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 2
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 2ca98f76a0f..f7a287a1d3b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -48,6 +48,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-14 09:02 Heikki Linnakangas <[email protected]>
parent: Evgeny Voropaev <[email protected]>
2 siblings, 2 replies; 17+ messages in thread
From: Heikki Linnakangas @ 2026-04-14 09:02 UTC (permalink / raw)
To: Evgeny Voropaev <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
On 24/03/2026 16:28, Evgeny Voropaev wrote:
> Also I hope, independently of its usage in prune/freeze records, the
> DFoR itself might be used for compression sequences in other places of PG.
Yeah, that would make this huge amount of new code much more palatable.
I had a similar thought when I added src/backend/lib/integerset.c, I
planned to also use it for holding the dead TID list in vacuum for
starters, and possibly for more things in the future. That plan was
foiled because we got parallel VACUUM instead, which moved the TID list
to shared memory, and I didn't account for that in integerset.c. So now
integerset.c is only used for GiST vacuum, which is a pretty narrow use
case.
Can this DFoR code replace integerset.c easily? Can we use it for the
vacuum dead TID list? For GIN posting lists? Where else?
- Heikki
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-14 09:11 Andrey Borodin <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 17+ messages in thread
From: Andrey Borodin @ 2026-04-14 09:11 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Evgeny Voropaev <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers
> On 14 Apr 2026, at 14:02, Heikki Linnakangas <[email protected]> wrote:
>
> Can this DFoR code replace integerset.c easily?
Or, possibly, teach integerset to be serializable to WAL record or anywhere else.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-21 05:41 Evgeny Voropaev <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 2 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-21 05:41 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
Hello hackers,
> Can this DFoR code replace integerset.c easily? Can we use it for
> the vacuum dead TID list? For GIN posting lists? Where else?
Heikki, thank you for your attention and proposals. I'm learning areas
you proposed to be developed. This took time, since I am not adept at
them. Last week I also have been developing the DFoR patch to support
unsorted sequences. That's why there was the delay in answering.
About GIN.
Since GIN exploits TIDs sequences and saves it on the disk, it can be
the most appropriate candidate to be developed with DFoR.
About the dead TID list.
If I'm not mistaken, the dead TID list exists only in RAM and never on
the disk or in the network. So, what is the advantage supposed to be
achieved due to using compression in the dead TID list?
About the GiST vacuuming and the use of integerset in it.
The integerset implements a tree in addition to compression.
DFoR now performs only compression. Moreover the size of a pack is
flexible (varying), which must become an issue for its usage in the
tree. It needs more thorough further elaboration to be developed.
So what do you think about improving GIN by means of DFOR? Should I try?
Best regards,
Evgeny Voropaev,
Tantor Labs, LLC.
P.S.
In the v12 version of the patch:
- implemented the DFOR compression for unsorted sequences;
- implemented the compression of frozen and redirected tuple offsets
in the prune/freeze WAL record
- ignored header checking of header templates from DFOR file set;
- rebased onto 9b43e6793b0.
Attachments:
[text/x-patch] v12-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (27.3K, 2-v12-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From 78f4a809885264ada485f8e9ae80e192e197d9b9 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:12:55 +0800
Subject: [PATCH v12 1/5] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allows developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 +++++++++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_templ.h | 14 ++
src/include/lib/bitpack_templ_staple.h | 57 +++++
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 +++
src/include/lib/vect_templ_staple.h | 140 ++++++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 11 +
src/include/lib/vect_u16_config.h | 10 +
src/tools/pginclude/headerscheck | 9 +
19 files changed, 810 insertions(+)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_staple.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..536fd604ab2
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_templ_staple.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 97ed8c63f5e..627bc331f02 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..208caf2b0e2
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_templ_staple.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_staple.h b/src/include/lib/bitpack_templ_staple.h
new file mode 100644
index 00000000000..c8ef3285004
--- /dev/null
+++ b/src/include/lib/bitpack_templ_staple.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_templ_staple.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_
+#define _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..453ceb6b416
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,11 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 785d6f867ad..e979d0a7436 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -158,6 +158,15 @@ do
test "$f" = src/interfaces/libpq/oauth-debug.h && continue
+ # These files are templates and cannot be included standalone
+ test "$f" = src/include/lib/bitpack_templ.h && continue
+ test "$f" = src/include/lib/bitpack_templ_staple.h && continue
+ test "$f" = src/include/lib/bitpack_templ_undef.h && continue
+
+ test "$f" = src/include/lib/vect_templ.h && continue
+ test "$f" = src/include/lib/vect_templ_staple.h && continue
+ test "$f" = src/include/lib/vect_templ_undef.h && continue
+
# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
--
2.53.0
[text/x-patch] v12-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch (69.2K, 3-v12-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch)
download | inline diff:
From 16ccc330eb329b52b085bd34da62eb779ca1faca Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:18:42 +0800
Subject: [PATCH v12 2/5] Tests of vect and uniqsortvect containers and of
bitpack algorithms.
Unit tests for the vect and the bitpack units are imlemented. Unit tests
are implemented as binary applications written in C language (ELF
executables) that support the TAP protocol and are run using the Prove
utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 54 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 ++++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 ++++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
20 files changed, 2088 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index cef1ad7f87d..0b6a22be18b 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..4fc9f4bc1ba
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,54 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# This fixes a problem in some CI jobs
+DEPDIR ?= ".deps"
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $< \
+ -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v12-0003-Implement-Delta-Frame-of-Reference-compression.patch (31.0K, 4-v12-0003-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 34e71cfef4afb316cd729a350cbf9ef5c60d9ec0 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:15:34 +0800
Subject: [PATCH v12 3/5] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
The current implementation of the DFOR algorithm supports items of
unsigned integer types. The type of an item is defined with the
item_t macro.
The compressed pack comprises three parts:
- deltas;
- exceptions positions.
- exceptions;
Each of them is a densily bit-packed sequence.
The delta is a difference between the current item and the previous one. The
delta of the first item (the item having the zero index) is its actual value:
delta[0] = m[0]-0 = m[0]. Serialised deltas is a sequence of bits. Each
serialised delta in 'deltas' has a fixed bit width. If the delta's width
exceeds the allowed size of a delta in 'deltas', the higher bits of this
delta is put into exceptions, its positions is saved into the 'excepton
positions' section. Usage of exceptions can be deliberately turned off.
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 722 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 126 +++++
src/include/lib/dfor_templ_undef.h | 28 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
9 files changed, 930 insertions(+)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..48280b2b9c1
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,722 @@
+/*
+ * dfor_templ.c
+ *
+ * Implement the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The current implementation of DFOR algorithm supports items of unsigned
+ * integer type. The type of original items is defined with the item_t macro.
+ *
+ * The compressed pack comprises three parts:
+ * - deltas (least significant bits of deltas);
+ * - exceptions positions.
+ * - exceptions (most significant bits of deltas);
+ * Each of them is a densily bit-packed sequence.
+ *
+ * The delta is a difference between the current item and the previous one. The
+ * delta of the first item (the item having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. A serialised delta is a sequence of bits. Each
+ * serialised delta in 'deltas' has a fixed bit width. If the delta's width
+ * exceeds some critical size, defined during analyzing, the higher bits of this
+ * delta is put into exceptions, which position is saved into the 'excepton
+ * positions' section. Usage of exceptions can be deliberately turned off.
+ *
+ * If signed deltas are used, the module of a delta is represented by most
+ * siginificant bits and the sign is saved as the least significant bit: 0 -
+ * delta is positive, 1 - delta is negative.
+ *
+ * DFoR supports both external memory (outer memory) provided by a caller
+ * and automatically managed memory, allocated by means of malloc, palloc
+ * or similar functions. Memory management configuration must be defined
+ * during initialization. All subsequent operations follow this
+ * configuration. For example, a caller can place a buffer on the stack to
+ * avoid heap allocation and pass the buffer to a DFoR unit. As a result,
+ * the packing and unpacking processes exclude dynamic allocation.
+ *
+ * The DFoR unit is implemented as a set of templates. Developers can
+ * generate DFoR implementations for any unsigned integer type (uint8_t,
+ * uint16_t, uint32_t, uint64_t).
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters,
+ bool *sign);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Do not calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ * The value of the signed_deltas argument can be changed by this function from
+ * false to true, but not vice versa. Once signed_deltas is set to true, it cannot
+ * be changed.
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters,
+ bool *signedDeltas)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ size_t maxItemWidth;
+ item_t maxItemMask;
+ item_t maxModulusOfDelta;
+
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL || usvDeltaWidths == NULL || signedDeltas == NULL)
+ return -1;
+
+ vect_clear(vDeltas);
+ vect_clear(usvDeltaWidths);
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+ else
+ vect_clear(vWidthCounters);
+
+ /*
+ * We use the maximum possible item width here which is equal ro the width
+ * of item_t. TODO: we can analyze items and calculate the real max item
+ * width. This must improve compression ratio.
+ */
+ maxItemWidth = sizeof(item_t) * 8;
+ maxItemMask = width_to_mask(maxItemWidth);
+ maxModulusOfDelta = (*signedDeltas) ? maxItemMask >> 1 : maxItemMask;
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++)
+ {
+ bool negDelta = arr[j] < prev;
+
+ if (negDelta)
+ {
+ if (unlikely(*signedDeltas == false))
+ { /* Recalculate all deltas as signed. */
+ *signedDeltas = true;
+ return dfor_calc_deltas(cnt, arr, vDeltas, usvDeltaWidths,
+ vWidthCounters, signedDeltas);
+ }
+ Assert(*signedDeltas == true);
+ delta = prev - arr[j];
+ }
+ else
+ delta = arr[j] - prev;
+
+ if (delta > maxModulusOfDelta)
+ {
+ Assert(*signedDeltas == true);
+ /* Use trick with overlapping here */
+ delta = maxItemMask - delta + 1;
+ negDelta = !negDelta;
+ }
+
+ Assert(delta <= maxModulusOfDelta);
+
+ if (*signedDeltas == true)
+ {
+ uint8 signCode = (negDelta) ? 1 : 0;
+ delta = delta << 1 | signCode;
+ width = (delta == 0) ? 2 : width_from_val(delta);
+ }
+ else
+ width = width_from_val(delta);
+
+ vect_append(vDeltas, delta);
+
+ if (vWidthCounters == NULL)
+ {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ }
+ else
+ {
+ insWidthInsert = usv_insert(usvDeltaWidths, width);
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ prev = arr[j];
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+ dfor->signed_deltas = false;
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL,
+ &dfor->signed_deltas))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t delta;
+ bool negDelta = false;
+ item_t prev = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+
+ size_t maxItemWidth;
+ item_t maxItemMask;
+
+ maxItemWidth = sizeof(item_t) * 8;
+ maxItemMask = width_to_mask(maxItemWidth);
+
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ delta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ delta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+
+ if(dfor->signed_deltas)
+ {
+ negDelta = delta & 0x01;
+ delta >>= 1;
+ if ((negDelta && prev < delta) ||
+ (!negDelta && delta > maxItemMask - prev))
+ {
+ delta = maxItemMask - delta + 1;
+ negDelta = !negDelta;
+ }
+ }
+
+ if (negDelta)
+ prev -= delta;
+ else
+ prev += delta;
+
+ vect_append(vVals, prev);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..8aa497dfaae
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters, bool *sign);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e47f6cb3a33
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,126 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+ bool signed_deltas;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..7a22ea9771f
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,28 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
--
2.53.0
[text/x-patch] v12-0004-Tests-for-Delta-Frame-of-Reference-unit.patch (23.4K, 5-v12-0004-Tests-for-Delta-Frame-of-Reference-unit.patch)
download | inline diff:
From 8a187541904fdfe614d194c10b765b11823ef237 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:18:07 +0800
Subject: [PATCH v12 4/5] Tests for Delta Frame of Reference unit.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 536 ++++++++++++++++++++++++++++++++++
4 files changed, 541 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 4fc9f4bc1ba..2ca98f76a0f 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -33,7 +33,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..efa91e3e6c8
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,536 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[],
+ bool deltaSinednessRequested,
+ bool deltaSinednessExpected);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[],
+ size_t cntDelta, uint16_t marDeltasExpected[],
+ size_t cntWidth, uint16_t marWidthsExpected[],
+ size_t cntStat, uint16_t marWidthsStatExpected[],
+ bool deltaSignednessRequested,
+ bool deltaSignednessExpected)
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+ bool deltaSignedness;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ deltaSignedness = deltaSignednessRequested;
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters, &deltaSignedness);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(deltaSignedness, "==", deltaSignednessExpected, "The signedness result is OK.");
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Deltas statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Deltas statistics statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean has been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(267);
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for sorted sequences \n");
+ {
+ /* Sorted sequences */
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1 }, /* marDeltasExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2, 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2 },
+ /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for sorted sequences PASSED \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for unsorted sequences and test SIGNEDNESS \n");
+
+ /* Unsorted sequences */
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0x0, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 10 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, +1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 7 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 2, 2, 2, 2, 2, 2, 2, 2, 3 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 11 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, -1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 7 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 2, 2, 2, 2, 2, 2, 2, 2, 3 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 11 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, -1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 3, 5, 7, 9, 11, 13, 15 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0x2, 0x4, 0x4, 0x5, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4 }, /* marDeltasExpected*/
+ /*BIN: 10, 100, 100, 101, 100, 100, 100, 100, 100, 100 */
+ /*DEC: +1, +2, +2, -2, +2, +2, +2, +2, +2, +2 */
+ 2, (uint16_t[]) { 2, 3 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 412 }, /* inArr */
+ 14,
+ (uint16_t[]) { 200, 200, 200, 200, 2, 2, 2, 2, 4, 4, 4, 4, 4, 5 }, /* marDeltasExpected code*/
+ /* DEC: +100, +100, +100, +1, +1, +1, +1, +1, +2, +2, +2, +2, +2, -2 */
+ 3, (uint16_t[]) { 2, 3, 8 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 246 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 8 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for unsorted sequences and test SIGNEDNESS PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for unsorted sequences and test OVERLAPPING trick \n");
+
+ test_delta_calculation(1, (uint16_t[]) { 33000 }, /* inArr */
+ 1, (uint16_t[]) { 33000 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 33000 }, /* inArr */
+ /* 65535 - 33000 + 1 = 32536 = 0x7F18
+ -32536 = (0x7F18 << 1) + 1 = 0xFE31 = 65073 */
+ 1, (uint16_t[]) { 0xFE31 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(2, (uint16_t[]) { 0, 33000 }, /* inArr */
+ 2, (uint16_t[]) { 0, 33000 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 16 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(2, (uint16_t[]) { 33000, 0 }, /* inArr */
+ /* 65535 - 33000 + 1 = 32536 = 0x7F18;
+ (0x7F18 << 1) = 0xFE30 = 65072;
+ -32536 = (0x7F18 << 1) | 1 = 0xFE31 = 65073 */
+ 2, (uint16_t[]) { 0xFE31, 0xFE30 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 2 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(16, (uint16_t[]) { 0, 1, 0, 3, 4, 5, 6, 7,
+ 8, 9, 10, 11, 12, 13, 14, 65534 }, /* inArr */
+ 16, (uint16_t[]) { 0x0, 0x2, 0x3, 0x6, 0x2, 0x2, 0x2, 0x2,
+ 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x21 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 2, 3, 6 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 14, 1, 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for unsorted sequences and test OVERLAPPING trick PASSED \n");
+ printf("========================================\n\n");
+ }
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics on widths */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ /* Unsorted sequences */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 13 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 2, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 32, /* awaited bits count */
+ 4, /* cntByteCountAwaited */
+ 7.99, /* awaited ratio min value (ratio >= 7.99) */
+ (uint8_t[]) { 0xA8, 0xAA, 0xAA, 0xEA });
+
+ /* Test the decoding algorithm for the overlap trick */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 65535, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 3, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 48, /* awaited bits count */
+ 6, /* cntByteCountAwaited */
+ 4, /* awaited ratio min value */
+ (uint8_t[]) { 0x50, 0x4d, 0x49, 0x92, 0x24, 0x49 });
+
+ /* The next test does not result in activation of the overlap trick since
+ the sequence is sorted. */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 65535 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 16, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 256, /* awaited bits count */
+ 32, /* cntByteCountAwaited */
+ 0.9999, /* awaited ratio min value = 1 */
+ (uint8_t[]) { 00, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 0xf1, 0xff });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v12-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (41.1K, 6-v12-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From 1fb7daa47decc6cf5cb33a1a1223a352060a9862 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v12 5/5] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 21 +-
src/backend/access/heap/pruneheap.c | 209 +++++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 114 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 94 ++++--
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 10 +-
src/include/access/heapam_xlog_dfor.h | 144 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 866 insertions(+), 59 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index 9ed7024e814..1d44cf96692 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,12 +106,26 @@ heap_xlog_prune_freeze(XLogReaderState *record)
OffsetNumber *frz_offsets;
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ char *cursor PG_USED_FOR_ASSERTS_ONLY;
- heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
+ /*
+ * Provide DFoR unpacking with outer buffers. 3 buffer parts are
+ * for saving the redirected, the dead and the unused tuple offsets.
+ * Additional three parts are for internal needs of the dfor_unpack
+ * function.
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[6 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
+ cursor = heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf_aligned.dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
@@ -156,7 +171,7 @@ heap_xlog_prune_freeze(XLogReaderState *record)
}
/* There should be no more data */
- Assert((char *) frz_offsets == dataptr + datalen);
+ Assert(cursor == dataptr + datalen);
/*
* The critical integrity requirement here is that we must never end
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..994e7b57c35 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,45 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t frozen_meta = {0};
+ dfor_meta_t redirected_meta = {0};
+ dfor_meta_t dead_meta = {0};
+ dfor_meta_t unused_meta = {0};
+
+ uint8 frozen_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 redirected_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 4 * DFOR_BUF_PART_SIZE
+ * - for 4 packed sequences: frozen, redirected, dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 7 * DFOR_BUF_PART_SIZE
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[7 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression &&
+ (nredirected > 5 || ndead > 9 || nunused > 9 || nfrozen > 9))
+ {
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+ }
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2674,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2640,39 +2696,138 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, plans,
sizeof(xlhp_freeze_plan) * nplans);
}
- if (nredirected > 0)
- {
- xlrec.flags |= XLHP_HAS_REDIRECTIONS;
- redirect_items.ntargets = nredirected;
- XLogRegisterBufData(0, &redirect_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, redirected,
- sizeof(OffsetNumber[2]) * nredirected);
- }
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int frozen_pack_res = 0;
+ int redirected_pack_res = 0;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
+
+ if (nfrozen > 0)
+ {
+ frozen_pack_res = dfor_u16_pack(nfrozen, frz_offsets,
+ DFOR_EXC_USE, &frozen_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ 3 * DFOR_BUF_PART_SIZE);
+ }
+
+ if (nredirected > 0)
+ {
+ redirected_pack_res = dfor_u16_pack(nredirected * 2, redirected,
+ DFOR_EXC_USE, &redirected_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf);
+ }
+
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ DFOR_BUF_PART_SIZE);
+ }
+
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ 2 * DFOR_BUF_PART_SIZE);
+ }
+
+ if (frozen_pack_res == 0 && redirected_pack_res == 0 &&
+ dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into the log. */
+ size_t meta_pack_sz;
+
+ if (nfrozen > 0)
+ {
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &frozen_meta, frozen_meta_pack);
+
+ XLogRegisterBufData(0, &frozen_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, frozen_meta.pack, frozen_meta.nbytes);
+ }
+ if (nredirected > 0)
+ {
+ xlrec.flags |= XLHP_HAS_REDIRECTIONS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &redirected_meta, redirected_meta_pack);
+
+ XLogRegisterBufData(0, &redirected_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, redirected_meta.pack,
+ redirected_meta.nbytes);
+ }
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (nfrozen > 0)
+ XLogRegisterBufData(0, frz_offsets,
+ sizeof(OffsetNumber) * nfrozen);
+ if (nredirected > 0)
+ {
+ xlrec.flags |= XLHP_HAS_REDIRECTIONS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ redirect_items.ntargets = nredirected;
+ XLogRegisterBufData(0, &redirect_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, redirected,
+ sizeof(OffsetNumber[2]) * nredirected);
+ }
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
- if (nfrozen > 0)
- XLogRegisterBufData(0, frz_offsets,
- sizeof(OffsetNumber) * nfrozen);
/*
* Prepare the main xl_heap_prune record. We already set the XLHP_HAS_*
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..bc38e4a2555
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,114 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->signed_deltas == 0) ? 0 : 1,
+ XLHPF_META_SIGNEDDELTAS_FLAG_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->signed_deltas == checker.signed_deltas);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ meta->signed_deltas = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_SIGNEDDELTAS_FLAG_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..a445b21ed36 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -102,17 +103,19 @@ plan_elem_desc(StringInfo buf, void *plan, void *data)
* This is in heapdesc.c so it can be shared between heap2_redo and heap2_desc
* code, the latter of which is used in frontend (pg_waldump) code.
*/
-void
+char *
heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
- xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *) cursor;
+ int nfrozen = 0;
+ xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *)cursor;
*nplans = freeze_plans->nplans;
Assert(*nplans > 0);
@@ -120,6 +123,20 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
cursor += offsetof(xlhp_freeze_plans, plans);
cursor += sizeof(xlhp_freeze_plan) * *nplans;
+
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, &nfrozen, frz_offsets,
+ dfor_buf);
+ }
+ else
+ {
+ for (int i = 0; i < *nplans; i++)
+ nfrozen += (*plans)[i].ntuples;
+
+ *frz_offsets = (OffsetNumber *) cursor;
+ cursor += sizeof(OffsetNumber) * nfrozen;
+ }
}
else
{
@@ -129,14 +146,25 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_REDIRECTIONS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, nredirected, redirected,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ Assert(*nredirected % 2 == 0);
+ *nredirected /= 2;
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+
+ *nredirected = subrecord->ntargets;
+ Assert(*nredirected > 0);
+ *redirected = &subrecord->data[0];
- *nredirected = subrecord->ntargets;
- Assert(*nredirected > 0);
- *redirected = &subrecord->data[0];
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber[2]) * *nredirected;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber[2]) * *nredirected;
+ }
}
else
{
@@ -146,14 +174,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf + 2 * DFOR_BUF_PART_SIZE);
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
}
else
{
@@ -163,22 +199,29 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if (flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + 3 * DFOR_BUF_PART_SIZE);
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
}
else
{
*nunused = 0;
*nowunused = NULL;
}
-
- *frz_offsets = (OffsetNumber *) cursor;
+ return cursor;
}
void
@@ -309,13 +352,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..2233ff7d108 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -489,11 +492,12 @@ extern const char *heap2_identify(uint8 info);
extern void heap_xlog_logical_rewrite(XLogReaderState *r);
/* in heapdesc.c, so it can be shared between frontend/backend code */
-extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
+extern char * heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..b169696e9bc
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,144 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage * 2. We
+ * multiply it by 2 since the sequence of redirected tuple offsets have a pair
+ * of offsets for each redirection and in the worst case we can have all tuples
+ * redirected.
+ *
+ * Since we can't calculate MaxHeapTuplesPerPage on preprocessor
+ * stage, we intentionally overestimate it as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count on a page = 1366. Redirected sequences can and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount) * 2.
+ * We mu
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|--------------------|--------|-----------------|
+ * | main | 0, 1 | 0-11 | item count | 12 bit | 1...2730 |
+ * | main | 1 | 12-15 | delta width | 4 bit | 1...12 |
+ * | main | 2 | 16 | signed deltas flag | 1 bit | 0...1 |
+ * | main | 2 | 17 | exception flag | 1 bit | 0...1 |
+ * | extra | 2, 3 | 18-26 | exception count | 9 bit | 1...274 |
+ * | extra | 3 | 27-29 | exception width | 4 bit | 1...11 |
+ * | extra | 3, 4 | 30-33 | except pos. width | 4 bit | 1...12 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 12
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_SIGNEDDELTAS_FLAG_SZ 1 /* Flag about signedness of deltas */
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 2ca98f76a0f..f7a287a1d3b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -48,6 +48,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-21 07:20 Heikki Linnakangas <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 0 replies; 17+ messages in thread
From: Heikki Linnakangas @ 2026-04-21 07:20 UTC (permalink / raw)
To: Evgeny Voropaev <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Andrey Borodin <[email protected]>
On 21/04/2026 08:41, Evgeny Voropaev wrote:
> Hello hackers,
>
>> Can this DFoR code replace integerset.c easily? Can we use it for
>> the vacuum dead TID list? For GIN posting lists? Where else?
>
> Heikki, thank you for your attention and proposals. I'm learning areas
> you proposed to be developed. This took time, since I am not adept at
> them. Last week I also have been developing the DFoR patch to support
> unsorted sequences. That's why there was the delay in answering.
>
> About GIN.
> Since GIN exploits TIDs sequences and saves it on the disk, it can be
> the most appropriate candidate to be developed with DFoR.
+1. And maybe the tid lists in to B-tree tuples too while we're at it.
For GIN posting lists, one important property of the current compression
scheme is that removing TIDs never makes the list larger than the
original. That's important for VACUUM, see
https://github.com/postgres/postgres/blob/d3bba041543593eb5341683107d899734dc8e73e/src/backend/acces...
> About the dead TID list.
> If I'm not mistaken, the dead TID list exists only in RAM and never on
> the disk or in the network. So, what is the advantage supposed to be
> achieved due to using compression in the dead TID list?
Reduces memory usage. And if it's faster to lookup than the current data
structure, that too. I don't know if that works out.
> About the GiST vacuuming and the use of integerset in it.
> The integerset implements a tree in addition to compression.
> DFoR now performs only compression. Moreover the size of a pack is
> flexible (varying), which must become an issue for its usage in the
> tree. It needs more thorough further elaboration to be developed.
Hmm. The integerset is a sparse list of integers, just like Frame of
Reference. The tree inside it is just an implementation detail. I was
thinking that you could replace the whole tree with DFoR, but I suppose
you cannot do random lookups in a DFoR compressed list, so you'd still
need the tree.
- Heikki
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Compress prune/freeze records with Delta Frame of Reference algorithm
@ 2026-04-21 08:48 Evgeny Voropaev <[email protected]>
parent: Evgeny Voropaev <[email protected]>
1 sibling, 0 replies; 17+ messages in thread
From: Evgeny Voropaev @ 2026-04-21 08:48 UTC (permalink / raw)
To: pgsql-hackers
Continue fixing CI issues.
Rebased onto master's d3bba041543.
Attachments:
[text/x-patch] v13-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch (27.3K, 2-v13-0001-Implement-vect-and-uniqsortvect-containers-and-b.patch)
download | inline diff:
From 55b6eda70c51fa3cda950aa2300b2040b1c47565 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:12:55 +0800
Subject: [PATCH v13 1/5] Implement vect and uniqsortvect containers and
bitpack algorithms.
The vect container stores arrays of integers and provides a set of
algorithms implementing essential operations on the contained array,
such as initialization, appending, inserting, and clearing.
The uniqsortvect container is based on the vect type but assumes that
its elements are sorted and unique. In addition to the algorithms
provided by vect, uniqsortvect implements binary search and the
specialized insertion routine.
The containers support both external memory provided by a caller and
automatically managed memory using malloc, Postgres's palloc, or similar
allocation functions. A container's strategy regarding memory management
must be set at container initialization, and all subsequent operations
honor this configuration. For example, a caller can place a buffer on
the stack to avoid heap allocation and pass the buffer to a vector
instance, which results in the vector performs no dynamic allocation.
This commit also introduces the bitpack unit, which provides algorithms
for dense bit-level packing and unpacking. The bitpack unit does not
use dynamic memory.
Each unit (vect, bitpack) is implemented as a set of templates that
allows developers to generate specialized solutions for any integer type
(uint8, int8, uint16, int16, and so on). The units bitpack_u16 and
vect_u16 supporting the uint16_t type are also provided by this commit.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile | 5 +
src/backend/lib/Makefile.dfor | 5 +
src/backend/lib/bitpack_templ.c | 156 +++++++++++++
src/backend/lib/bitpack_u16.c | 8 +
src/backend/lib/meson.build | 7 +
src/backend/lib/vect_templ.c | 301 +++++++++++++++++++++++++
src/backend/lib/vect_u16.c | 8 +
src/include/c.h | 4 +
src/include/lib/bitpack_templ.h | 14 ++
src/include/lib/bitpack_templ_staple.h | 57 +++++
src/include/lib/bitpack_templ_undef.h | 5 +
src/include/lib/bitpack_u16.h | 12 +
src/include/lib/bitpack_u16_config.h | 6 +
src/include/lib/vect_templ.h | 27 +++
src/include/lib/vect_templ_staple.h | 140 ++++++++++++
src/include/lib/vect_templ_undef.h | 25 ++
src/include/lib/vect_u16.h | 11 +
src/include/lib/vect_u16_config.h | 10 +
src/tools/pginclude/headerscheck | 9 +
19 files changed, 810 insertions(+)
create mode 100644 src/backend/lib/Makefile.dfor
create mode 100644 src/backend/lib/bitpack_templ.c
create mode 100644 src/backend/lib/bitpack_u16.c
create mode 100644 src/backend/lib/vect_templ.c
create mode 100644 src/backend/lib/vect_u16.c
create mode 100644 src/include/lib/bitpack_templ.h
create mode 100644 src/include/lib/bitpack_templ_staple.h
create mode 100644 src/include/lib/bitpack_templ_undef.h
create mode 100644 src/include/lib/bitpack_u16.h
create mode 100644 src/include/lib/bitpack_u16_config.h
create mode 100644 src/include/lib/vect_templ.h
create mode 100644 src/include/lib/vect_templ_staple.h
create mode 100644 src/include/lib/vect_templ_undef.h
create mode 100644 src/include/lib/vect_u16.h
create mode 100644 src/include/lib/vect_u16_config.h
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index b6cefd9cca0..74167bc9e4c 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -12,6 +12,8 @@ subdir = src/backend/lib
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
+include Makefile.dfor
+
OBJS = \
bipartite_match.o \
bloomfilter.o \
@@ -22,5 +24,8 @@ OBJS = \
knapsack.o \
pairingheap.o \
rbtree.o \
+ $(OBJS_DFOR) \
+
+CPPFLAGS := -I$(top_srcdir)/src/backend/lib $(CPPFLAGS)
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
new file mode 100644
index 00000000000..b93c6e78644
--- /dev/null
+++ b/src/backend/lib/Makefile.dfor
@@ -0,0 +1,5 @@
+# Makefile.dfor
+
+OBJS_DFOR := \
+ bitpack_u16.o \
+ vect_u16.o
diff --git a/src/backend/lib/bitpack_templ.c b/src/backend/lib/bitpack_templ.c
new file mode 100644
index 00000000000..536fd604ab2
--- /dev/null
+++ b/src/backend/lib/bitpack_templ.c
@@ -0,0 +1,156 @@
+/*
+ * bitpack_templ.c
+ *
+ * The BITPACK unit implements routines pertaining to bit-packing. The bitpack
+ * unit allow higher-level functions to create high-density arrays packed
+ * bit-by-bit. In general, width of each item in a bitpacked array can vary and
+ * have not to be of fixed size, items can have different length.
+ */
+
+#include "lib/bitpack_templ_staple.h"
+
+item_t width_from_val(item_t val);
+item_t width_to_mask(size_t width);
+size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+/*
+ * Since width of item_t cannot be more than length of item_t
+ * lg(MAX(item_t))+1, we use the item_t type for returned value
+ */
+item_t
+width_from_val(item_t val)
+{
+ item_t width = 0;
+
+ while (val) {
+ width++;
+ val = val >> 1;
+ }
+
+ return width == 0 ? 1 : width;
+}
+
+item_t
+width_to_mask(size_t width)
+{
+ size_t mask = 0;
+
+ Assert(width != 0);
+ Assert(width <= sizeof(item_t) * 8);
+
+ if (likely(width < sizeof(size_t)))
+ mask = (1 << width) - 1;
+ else
+ while (width--)
+ mask = (mask << 1) | 1;
+
+ return (item_t)mask;
+}
+
+size_t
+bitpack_pack(uint8_t *pack, size_t caret, item_t item, size_t szItemWidth)
+{
+ size_t szItemWidthToGo = szItemWidth;
+ item_t itmMaskToGo = width_to_mask(szItemWidth);
+
+ while (szItemWidthToGo > 0) {
+ size_t cntSavedBits;
+ size_t byte = caret / 8;
+ size_t off = caret % 8;
+ uint8_t ubChunk = (uint8_t)item << off;
+ item_t itmChunkMask = itmMaskToGo << off;
+ /*
+ * Applying chunk using the mask. Setting bits to one and resetting bits
+ * to zero is only in scopes defined by the mask. Zeroing of bits
+ * according to a mask, we can use even a pack not been nulled in
+ * advance.
+ */
+ pack[byte] |= (ubChunk & itmChunkMask);
+ pack[byte] &= (ubChunk | ~itmChunkMask);
+ cntSavedBits = (8 - off > szItemWidthToGo) ?
+ szItemWidthToGo :
+ 8 - off; // number of saved bits
+ szItemWidthToGo -= cntSavedBits;
+ caret += cntSavedBits;
+ item = item >> cntSavedBits;
+ itmMaskToGo = itmMaskToGo >> cntSavedBits;
+ }
+ return caret;
+}
+
+item_t
+bitpack_unpack(const uint8_t *pack, size_t *caret, size_t widItem)
+{
+ size_t szItemCaret;
+ size_t szItemWidthToGo;
+ uint8_t item[sizeof(item_t)]; /* size of item array */
+
+ size_t szPackByte;
+ size_t szPackOff;
+ size_t szItemByte;
+ size_t szItemOff;
+ uint8_t ubChunk;
+
+ szItemCaret = 0;
+ szItemWidthToGo = widItem;
+ memset(item, 0, sizeof(item_t));
+
+ while (szItemWidthToGo > 0) {
+ size_t szChunkSize;
+ size_t szChunkLowSize, szChunkHighSize;
+
+ szPackByte = *caret / 8;
+ szPackOff = *caret % 8;
+ szItemByte = szItemCaret / 8;
+ szItemOff = szItemCaret % 8;
+
+ ubChunk = pack[szPackByte] >> szPackOff;
+
+ szChunkSize = 8 - szPackOff;
+ if (szItemWidthToGo < szChunkSize) {
+ szChunkSize = szItemWidthToGo;
+ ubChunk = ubChunk & (uint8_t)width_to_mask(szItemWidthToGo);
+ }
+
+ if (szChunkSize > (8 - szItemOff)) /* Free space of item[szItemByte] */
+ {
+ szChunkLowSize = 8 - szItemOff;
+ szChunkHighSize = szChunkSize - szChunkLowSize;
+ } else {
+ szChunkLowSize = szChunkSize;
+ szChunkHighSize = 0;
+ }
+
+ item[szItemByte] |= ubChunk << szItemOff; /* chunk_low */
+
+ if (szChunkHighSize != 0) {
+ Assert((szItemByte + 1) < sizeof(item_t)); /* size of item array */
+ item[szItemByte + 1] |= ubChunk >> szChunkLowSize; /* chunk_high */
+ }
+
+ *caret += szChunkSize;
+ szItemCaret += szChunkSize;
+ szItemWidthToGo -= szChunkSize;
+ }
+
+ /*
+ * Reordering bytes in accordance with endianness of the system.
+ *
+ * Here for a Little-endian system we can avoid reordering, but in such a
+ * case we need to keep the item array aligned with item_t type, but we do
+ * not keep.
+ */
+ {
+ size_t j = 1;
+ item_t val = item[sizeof(item_t) - j];
+ while (++j <= sizeof(item_t)) {
+ val = val << 8;
+ val |= item[sizeof(item_t) - j];
+ }
+ return val;
+ }
+}
+
+#include "lib/bitpack_templ_undef.h"
diff --git a/src/backend/lib/bitpack_u16.c b/src/backend/lib/bitpack_u16.c
new file mode 100644
index 00000000000..ae2ee6d6bb2
--- /dev/null
+++ b/src/backend/lib/bitpack_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: bitpack_u16.c
+ */
+
+/* clang-format off */
+#include "lib/bitpack_u16_config.h"
+#include "bitpack_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 8e38fb20f17..0984bd0e3f6 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -1,5 +1,10 @@
# Copyright (c) 2022-2026, PostgreSQL Global Development Group
+dfor_sources = files(
+ 'bitpack_u16.c',
+ 'vect_u16.c'
+)
+
backend_sources += files(
'bipartite_match.c',
'bloomfilter.c',
@@ -11,3 +16,5 @@ backend_sources += files(
'pairingheap.c',
'rbtree.c',
)
+
+backend_sources += dfor_sources
diff --git a/src/backend/lib/vect_templ.c b/src/backend/lib/vect_templ.c
new file mode 100644
index 00000000000..52713c39d3b
--- /dev/null
+++ b/src/backend/lib/vect_templ.c
@@ -0,0 +1,301 @@
+/*
+ * File: vect_templ.c
+ */
+
+#include "lib/vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+/*
+ * Caller has to control whether vector use outer memory provided by caller or
+ * manage memory allocation automatically, which defines whether vect_insert,
+ * vect_append and other functions of the vector container automatically mange
+ * dynamic memory allocation or not.
+ */
+
+int vect_init(vect_t *v, size_t cap, item_t outer_mem[]);
+int vect_fill(vect_t *v, size_t cnt, const item_t in[]);
+int vect_reserve(vect_t *v, size_t szNewCap);
+int vect_append(vect_t *vect, item_t val);
+void vect_print(const vect_t *a);
+int vect_compare(const vect_t *a, const vect_t *b);
+int vect_insert(vect_t *v, size_t pos, item_t val);
+void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.h
+ */
+usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+int
+vect_init(vect_t *v, size_t cap, item_t outer_mem[])
+{
+ if (v == NULL)
+ goto vect_init_error;
+
+ v->cap = cap;
+ v->cnt = 0;
+
+ if (outer_mem != NULL)
+ {
+ v->mem_is_outer = true;
+ v->m = outer_mem;
+ }
+ else
+ {
+ v->mem_is_outer = false;
+ if (cap == 0)
+ v->m = NULL;
+ else
+ {
+ v->m = (item_t *)VECT_MALLOC(cap * sizeof(item_t));
+ if (v->m == NULL)
+ goto vect_init_error;
+ }
+ }
+
+ /* vect_init_ok: */
+ return 0;
+vect_init_error:
+ memset(v, 0, sizeof(vect_t));
+ return -1;
+}
+
+int
+vect_fill(vect_t *v, size_t cnt, const item_t in[])
+{
+ if (v == NULL)
+ return -1;
+
+ if (cnt == 0)
+ {
+ vect_clear(v);
+ return 0;
+ }
+
+ for (size_t j = 0; j < cnt; j++)
+ {
+ if (vect_append(v, in[j]) != 0)
+ {
+ vect_clear(v);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int
+vect_reserve(vect_t *v, size_t szNewCap)
+{
+ item_t *mNew;
+
+ if (v == NULL)
+ return -1;
+
+ if (v->mem_is_outer)
+ return -1;
+
+ if (szNewCap <= v->cap)
+ return 0;
+
+ mNew = (item_t *) VECT_MALLOC(sizeof(item_t) * szNewCap);
+
+ if (mNew == NULL)
+ return -1;
+
+ if(v->m == NULL && v->cnt != 0)
+ return -1;
+
+ if(v->m != NULL && v->cnt != 0)
+ memcpy(mNew, v->m, v->cnt * sizeof(item_t));
+
+ VECT_FREE(v->m);
+ v->m = mNew;
+ v->cap = szNewCap;
+ return 0;
+}
+
+int
+vect_append(vect_t *vect, item_t val)
+{
+ if (vect == NULL)
+ return -1;
+
+ if (vect->cnt + 1 > vect->cap)
+ {
+ if (vect->mem_is_outer)
+ return -1;
+ else
+ vect_reserve(vect, vect->cap + VECT_MEMALLOCSTEP);
+ }
+
+ vect->m[vect->cnt] = val;
+ vect->cnt++;
+ return 0;
+}
+
+void
+vect_print(const vect_t *a)
+{
+ for (size_t j = 0; j < a->cnt; j++)
+ printf("%" VECT_ITEM_FORMAT_SPECIFIER " ", a->m[j]);
+
+ printf("\n");
+}
+
+int
+vect_compare(const vect_t *a, const vect_t *b)
+{
+ if (a == NULL || b == NULL)
+ return -1;
+
+ if (a->cnt != b->cnt)
+ return -1;
+
+ for (size_t j = 0; j < a->cnt; j++)
+ if (a->m[j] != b->m[j])
+ return -1;
+
+ return 0;
+}
+
+int
+vect_insert(vect_t *v, size_t pos, item_t val)
+{
+ if (v->cap < v->cnt + 1 &&
+ (v->mem_is_outer || vect_reserve(v, v->cap + VECT_MEMALLOCSTEP) != 0))
+ return -1;
+
+ /*
+ * If need, move right from pos including pos. Because
+ * neither stdlib's nor POSIX's documentation defines the
+ * behaviour of memmove in case of count=0, we check it by
+ * ourselves.
+ */
+ if (v->cnt - pos > 0)
+ memmove(&v->m[pos + 1], &v->m[pos], (v->cnt - pos) * sizeof(item_t));
+
+ v->m[pos] = val;
+ v->cnt++;
+ return 0;
+}
+
+void
+vect_clear(vect_t *v)
+{
+ if (v == NULL)
+ return;
+
+ if (!v->mem_is_outer)
+ VECT_FREE(v->m);
+
+ memset(v, 0, sizeof(vect_t));
+}
+
+usv_srch_res_t
+usv_search(const uniqsortvect_t *usv, item_t val)
+{
+ size_t i, l, g;
+ usv_srch_res_t res;
+
+ if (usv == NULL || (usv->m == NULL && ((usv->cnt != 0) || usv->cap != 0))) {
+ res.st = USV_SRCH_ERROR;
+ return res;
+ }
+
+ if (usv->cnt == 0) {
+ res.pos = 0;
+ res.st = USV_SRCH_EMPTY;
+ return res;
+ }
+
+ if (val < usv->m[0]) {
+ res.pos = 0;
+ res.st = USV_SRCH_NOT_FOUND_SMALLEST;
+ return res;
+ }
+
+ if (val > usv->m[usv->cnt - 1]) {
+ res.pos = usv->cnt - 1;
+ res.st = USV_SRCH_NOT_FOUND_LARGEST;
+ return res;
+ }
+
+ l = 0;
+ g = usv->cnt - 1;
+
+ while (g - l > 1) {
+ i = l + (g - l) / 2;
+ if (val == usv->m[i]) {
+ res.pos = i;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val > usv->m[i]) {
+ l = i;
+ } else // val <= usv->m[i]
+ {
+ g = i;
+ }
+ }
+ /*
+ * When scopes l and g are neighbours ( g-l = 1)
+ */
+ if (val == usv->m[g]) {
+ res.pos = g;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ } else if (val == usv->m[l]) {
+ res.pos = l;
+ res.st = USV_SRCH_FOUND;
+ return res;
+ }
+
+ res.pos = g;
+ res.st = USV_SRCH_NOT_FOUND;
+ return res;
+}
+
+/*
+ * INSERT
+ * receives a value, checks whether an unique sorted values vector contains
+ * this value. If not, inserts new value, retaining sorted order.
+ */
+usv_ins_res_t
+usv_insert(uniqsortvect_t *a, item_t val)
+{
+ usv_srch_res_t search;
+ usv_ins_res_t insert;
+
+ Assert(a != NULL);
+
+ search = usv_search(a, val);
+ if (search.st == USV_SRCH_FOUND) {
+ insert.st = USV_INS_EXISTS;
+ insert.pos = search.pos;
+ return insert;
+ } else if (search.st == USV_SRCH_NOT_FOUND_SMALLEST ||
+ search.st == USV_SRCH_NOT_FOUND) {
+ insert.pos = search.pos;
+ } else if (search.st == USV_SRCH_EMPTY ||
+ search.st == USV_SRCH_NOT_FOUND_LARGEST) {
+ /* In case when value is more than largest: pos = a->cnt = search.g + 1.
+ */
+ /* In case of empty vector: pos = a->cnt = 0. */
+ insert.pos = a->cnt;
+ } else /* USV_SRCH_ERROR or unknown result */
+ {
+ insert.st = USV_INS_ERROR;
+ return insert;
+ }
+
+ insert.st = vect_insert(a, insert.pos, val)
+ == 0 ? USV_INS_NEW : USV_INS_ERROR;
+
+ return insert;
+}
+
+#include "lib/vect_templ_undef.h"
diff --git a/src/backend/lib/vect_u16.c b/src/backend/lib/vect_u16.c
new file mode 100644
index 00000000000..0ab8e224c7a
--- /dev/null
+++ b/src/backend/lib/vect_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: vect_u16.c
+ */
+
+/* clang-format off */
+#include "lib/vect_u16_config.h"
+#include "vect_templ.c"
+/* clang-format on */
diff --git a/src/include/c.h b/src/include/c.h
index 97ed8c63f5e..627bc331f02 100644
--- a/src/include/c.h
+++ b/src/include/c.h
@@ -505,6 +505,10 @@ extern "C++"
#define CppAsString(identifier) #identifier
#define CppAsString2(x) CppAsString(x)
#define CppConcat(x, y) x##y
+#define CppConcat2(x, y) CppConcat(x, y)
+
+#define CppConcatTriple(x, y, z) x##y##z
+#define CppConcatTriple2(a, b, c) CppConcatTriple(a, b, c)
/*
* VA_ARGS_NARGS
diff --git a/src/include/lib/bitpack_templ.h b/src/include/lib/bitpack_templ.h
new file mode 100644
index 00000000000..208caf2b0e2
--- /dev/null
+++ b/src/include/lib/bitpack_templ.h
@@ -0,0 +1,14 @@
+/*
+ * bitpack_templ.h
+ *
+ */
+
+#include "bitpack_templ_staple.h"
+
+extern item_t width_from_val(item_t val);
+extern item_t width_to_mask(size_t width);
+extern size_t bitpack_pack(uint8_t *pack, size_t caret, item_t item,
+ size_t szItemWidth);
+extern item_t bitpack_unpack(const uint8_t *pack, size_t *caret, size_t szItemWidth);
+
+#include "bitpack_templ_undef.h"
diff --git a/src/include/lib/bitpack_templ_staple.h b/src/include/lib/bitpack_templ_staple.h
new file mode 100644
index 00000000000..c8ef3285004
--- /dev/null
+++ b/src/include/lib/bitpack_templ_staple.h
@@ -0,0 +1,57 @@
+/*
+ * File: bitpack_templ_staple.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_
+#define _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_
+
+/* No code here yet */
+
+#endif /* _BITPACK_TEMPL_STAPLE_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if BITPACK_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef BITPACK_ITEM_TYPE
+#error "BITPACK_ITEM_TYPE macro is indefined."
+#endif
+#ifndef BITPACK_MARKER
+#error "BITPACK_MARKER macro is indefined."
+#endif
+
+#define item_t BITPACK_ITEM_TYPE
+#define width_from_val CppConcatTriple2(width_, BITPACK_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, BITPACK_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, BITPACK_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, BITPACK_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ *
+ * #include "lib/bitpack_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/bitpack_templ_undef.h b/src/include/lib/bitpack_templ_undef.h
new file mode 100644
index 00000000000..5bf864ffa15
--- /dev/null
+++ b/src/include/lib/bitpack_templ_undef.h
@@ -0,0 +1,5 @@
+#undef item_t
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/bitpack_u16.h b/src/include/lib/bitpack_u16.h
new file mode 100644
index 00000000000..45fb6c4b17b
--- /dev/null
+++ b/src/include/lib/bitpack_u16.h
@@ -0,0 +1,12 @@
+/*
+ * bitpack.h
+ */
+#ifndef _BITPACK_U16_H_
+#define _BITPACK_U16_H_
+
+/* clang-format off */
+#include "bitpack_u16_config.h"
+#include "bitpack_templ.h"
+/* clang-format on */
+
+#endif /* _BITPACK_U16_H_ */
diff --git a/src/include/lib/bitpack_u16_config.h b/src/include/lib/bitpack_u16_config.h
new file mode 100644
index 00000000000..9e6c64d4fee
--- /dev/null
+++ b/src/include/lib/bitpack_u16_config.h
@@ -0,0 +1,6 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define BITPACK_ITEM_TYPE uint16_t
+#define BITPACK_MARKER u16
diff --git a/src/include/lib/vect_templ.h b/src/include/lib/vect_templ.h
new file mode 100644
index 00000000000..8eec6f064b3
--- /dev/null
+++ b/src/include/lib/vect_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: vect_templ.h
+ */
+
+#include "vect_templ_staple.h"
+
+/*
+ * Vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern int vect_init(vect_t *v, size_t cap, item_t external_memory[]);
+extern int vect_fill(vect_t *v, size_t cnt, const item_t *in);
+extern int vect_reserve(vect_t *v, size_t szNewCap);
+extern int vect_append(vect_t *vect, item_t val);
+extern void vect_print(const vect_t *a);
+extern int vect_compare(const vect_t *a, const vect_t *b);
+extern int vect_insert(vect_t *v, size_t pos, item_t val);
+extern void vect_clear(vect_t *v);
+
+/*
+ * Unique sorted vector's functions
+ * Keep this section equal to the same section in vect_templ.c
+ */
+extern usv_ins_res_t usv_insert(uniqsortvect_t *a, item_t val);
+extern usv_srch_res_t usv_search(const uniqsortvect_t *usv, item_t val);
+
+#include "vect_templ_undef.h"
\ No newline at end of file
diff --git a/src/include/lib/vect_templ_staple.h b/src/include/lib/vect_templ_staple.h
new file mode 100644
index 00000000000..b192c6d82a3
--- /dev/null
+++ b/src/include/lib/vect_templ_staple.h
@@ -0,0 +1,140 @@
+/*
+ * File: vect_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+/*
+ * SEARCH in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_SRCH_ERROR = -1,
+ USV_SRCH_FOUND = 0,
+ USV_SRCH_EMPTY,
+ USV_SRCH_NOT_FOUND,
+ USV_SRCH_NOT_FOUND_SMALLEST,
+ USV_SRCH_NOT_FOUND_LARGEST
+} usv_srch_stat_t;
+
+typedef struct
+{
+ usv_srch_stat_t st;
+ size_t pos; /* position (index) of a member that is equal to searched value
+ * or that is nearest greater member */
+} usv_srch_res_t;
+
+/*
+ * INSERT in Vector of Unique Sorted members
+ */
+typedef enum
+{
+ USV_INS_ERROR = -1,
+ USV_INS_EXISTS = 0,
+ USV_INS_NEW
+} usv_ins_stat_t;
+
+typedef struct
+{
+ usv_ins_stat_t st;
+ size_t pos; /* position (index) of a member that was inserted or that proved
+ * to be equal to inserted value
+ */
+} usv_ins_res_t;
+
+#endif /* _VECT_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if VECT_MARKER
+ * redefined. This allow creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef VECT_ITEM_TYPE
+#error "VECT_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef VECT_ITEM_FORMAT_SPECIFIER
+#error "VECT_ITEM_FORMAT_SPECIFIER macro is indefined."
+#endif
+
+#ifndef VECT_MARKER
+#error "VECT_MARKER macro is indefined."
+#endif
+
+#ifndef VECT_MEMALLOCSTEP
+#error "VECT_MEMALLOCSTEP macro is indefined."
+#endif
+
+#ifndef VECT_MALLOC
+#error "VECT_MALLOC macro is indefined."
+#endif
+
+#ifndef VECT_FREE
+#error "VECT_FREE macro is indefined."
+#endif
+
+/*
+ * The Vector type itself,
+ * The Vector of Unique Sorted Items type
+ * and the Item type
+ *
+ * In fact, vectors's names looks like vect_u16_t where:
+ * vect_ - common prefix,
+ * u16 - marker,
+ * _t - suffix
+ */
+#define vect_t CppConcatTriple2(vect_, VECT_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, VECT_MARKER, _t)
+#define item_t VECT_ITEM_TYPE
+
+typedef struct
+{
+ size_t cnt; /* number of items */
+ size_t cap; /* capacity */
+ bool mem_is_outer; /* flag about an external memory is used */
+ item_t *m; /* items (members) */
+} vect_t;
+
+typedef vect_t uniqsortvect_t;
+
+#define vect_init CppConcatTriple2(vect_, VECT_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, VECT_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, VECT_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, VECT_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, VECT_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, VECT_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, VECT_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, VECT_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, VECT_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, VECT_MARKER, _search)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include
+ * #include "vect_templ_undef.h"
+ * in your file that uses this header
+ *
+ */
diff --git a/src/include/lib/vect_templ_undef.h b/src/include/lib/vect_templ_undef.h
new file mode 100644
index 00000000000..59b69f18b99
--- /dev/null
+++ b/src/include/lib/vect_templ_undef.h
@@ -0,0 +1,25 @@
+/*
+ * File: vect_undef.h
+ */
+
+#undef vect_t
+#undef uniqsortvect_t
+#undef item_t
+
+#undef VECT_ITEM_TYPE
+#undef VECT_MARKER
+#undef VECT_CppConcatTriple2
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef VECT_PRI_FORMAT_SPECIFIER
\ No newline at end of file
diff --git a/src/include/lib/vect_u16.h b/src/include/lib/vect_u16.h
new file mode 100644
index 00000000000..453ceb6b416
--- /dev/null
+++ b/src/include/lib/vect_u16.h
@@ -0,0 +1,11 @@
+/*
+ * File: vect_u16.h
+ */
+
+#ifndef _VECT_U16_H_
+#define _VECT_U16_H_
+
+#include "vect_u16_config.h"
+#include "vect_templ.h"
+
+#endif //_VECT_U16_H_
diff --git a/src/include/lib/vect_u16_config.h b/src/include/lib/vect_u16_config.h
new file mode 100644
index 00000000000..13a93284e8f
--- /dev/null
+++ b/src/include/lib/vect_u16_config.h
@@ -0,0 +1,10 @@
+/*
+ * File: vect_u16_config.h
+ */
+
+#define VECT_ITEM_TYPE uint16_t
+#define VECT_ITEM_FORMAT_SPECIFIER PRIu16
+#define VECT_MARKER u16
+#define VECT_MEMALLOCSTEP 5
+#define VECT_MALLOC malloc
+#define VECT_FREE free
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 785d6f867ad..e979d0a7436 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -158,6 +158,15 @@ do
test "$f" = src/interfaces/libpq/oauth-debug.h && continue
+ # These files are templates and cannot be included standalone
+ test "$f" = src/include/lib/bitpack_templ.h && continue
+ test "$f" = src/include/lib/bitpack_templ_staple.h && continue
+ test "$f" = src/include/lib/bitpack_templ_undef.h && continue
+
+ test "$f" = src/include/lib/vect_templ.h && continue
+ test "$f" = src/include/lib/vect_templ_staple.h && continue
+ test "$f" = src/include/lib/vect_templ_undef.h && continue
+
# We can't make these Bison output files compilable standalone
# without using "%code require", which old Bison versions lack.
# parser/gram.h will be included by parser/gramparse.h anyway.
--
2.53.0
[text/x-patch] v13-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch (69.2K, 3-v13-0002-Tests-of-vect-and-uniqsortvect-containers-and-of.patch)
download | inline diff:
From c81a333fb5b0c37616c0ebf73f21cd1c4b21abb3 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 20:18:42 +0800
Subject: [PATCH v13 2/5] Tests of vect and uniqsortvect containers and of
bitpack algorithms.
Unit tests for the vect and the bitpack units are imlemented. Unit tests
are implemented as binary applications written in C language (ELF
executables) that support the TAP protocol and are run using the Prove
utility.
The new Makefile target, check-unit, is integrated into the PostgreSQL
build system and allows running the unit tests using the command 'make
check-unit'.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
GNUmakefile.in | 1 +
src/Makefile.global.in | 2 +-
src/test/Makefile | 1 +
src/test/dfor/.gitignore | 3 +
src/test/dfor/Makefile | 54 ++++
src/test/dfor/meson.build | 62 ++++
src/test/dfor/test.h | 31 ++
src/test/dfor/test_bitpack_u16.c | 357 ++++++++++++++++++++++
src/test/dfor/test_uniqsortvect_u16.c | 263 ++++++++++++++++
src/test/dfor/test_vect_u16.c | 168 ++++++++++
src/test/libtap/.gitignore | 13 +
src/test/libtap/.travis.yml | 13 +
src/test/libtap/COPYING | 165 ++++++++++
src/test/libtap/INSTALL | 41 +++
src/test/libtap/Makefile | 73 +++++
src/test/libtap/Makefile.win | 37 +++
src/test/libtap/README.md | 268 ++++++++++++++++
src/test/libtap/tap.c | 421 ++++++++++++++++++++++++++
src/test/libtap/tap.h | 115 +++++++
src/test/meson.build | 1 +
20 files changed, 2088 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/.gitignore
create mode 100644 src/test/dfor/Makefile
create mode 100644 src/test/dfor/meson.build
create mode 100644 src/test/dfor/test.h
create mode 100644 src/test/dfor/test_bitpack_u16.c
create mode 100644 src/test/dfor/test_uniqsortvect_u16.c
create mode 100644 src/test/dfor/test_vect_u16.c
create mode 100644 src/test/libtap/.gitignore
create mode 100644 src/test/libtap/.travis.yml
create mode 100644 src/test/libtap/COPYING
create mode 100644 src/test/libtap/INSTALL
create mode 100644 src/test/libtap/Makefile
create mode 100644 src/test/libtap/Makefile.win
create mode 100644 src/test/libtap/README.md
create mode 100644 src/test/libtap/tap.c
create mode 100644 src/test/libtap/tap.h
diff --git a/GNUmakefile.in b/GNUmakefile.in
index cf6e759486e..3d9a42d6ad4 100644
--- a/GNUmakefile.in
+++ b/GNUmakefile.in
@@ -69,6 +69,7 @@ check check-tests installcheck installcheck-parallel installcheck-tests: submake
$(MAKE) -C src/test/regress $@
$(call recurse,check-world,src/test src/pl src/interfaces contrib src/bin src/tools/pg_bsd_indent,check)
+$(call recurse,check-unit,src/test,check-unit)
$(call recurse,checkprep, src/test src/pl src/interfaces contrib src/bin)
$(call recurse,installcheck-world,src/test src/pl src/interfaces contrib src/bin,installcheck)
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index cef1ad7f87d..0b6a22be18b 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -19,7 +19,7 @@
#
# Meta configuration
-standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck init-po update-po
+standard_targets = all install installdirs uninstall clean distclean coverage check checkprep installcheck check-unit init-po update-po
# these targets should recurse even into subdirectories not being built:
standard_always_targets = clean distclean
diff --git a/src/test/Makefile b/src/test/Makefile
index 3eb0a06abb4..aba8db1f483 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -14,6 +14,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = \
authentication \
+ dfor \
isolation \
modules \
perl \
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
new file mode 100644
index 00000000000..0d77a51216b
--- /dev/null
+++ b/src/test/dfor/.gitignore
@@ -0,0 +1,3 @@
+test_bitpack_u16
+test_uniqsortvect_u16
+test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
new file mode 100644
index 00000000000..4fc9f4bc1ba
--- /dev/null
+++ b/src/test/dfor/Makefile
@@ -0,0 +1,54 @@
+#-------------------------------------------------------------------------
+# File: src/test/dfor/Makefile
+#-------------------------------------------------------------------------
+
+subdir = src/test/dfor
+top_builddir = ../../..
+dfor_dir := $(top_builddir)/src/backend/lib
+
+include $(dfor_dir)/Makefile.dfor
+
+# Ensure dependency tracking works
+OBJS += $(OBJS_DFOR)
+
+include $(top_builddir)/src/Makefile.global
+
+# This fixes a problem in some CI jobs
+DEPDIR ?= ".deps"
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this Makefile. Tests don't use ones from src/backend/lib and compile
+# different ones for themselves.
+$(info Use OBJS_DFOR=$(OBJS_DFOR) from current directory $(subdir). \
+ They are built on sources from $(dfor_dir))
+
+$(OBJS_DFOR): %.o: $(dfor_dir)/%.c
+ @if test ! -d $(DEPDIR); then mkdir -p $(DEPDIR); fi
+ $(CC) $(CFLAGS) $(CPPFLAGS) -DFRONTEND \
+ -c $< \
+ -MMD -MP -MF $(DEPDIR)/$(notdir $<:.c=.Po) \
+ -o $@
+
+LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
+
+TESTS= test_vect_u16 \
+ test_uniqsortvect_u16 \
+ test_bitpack_u16
+
+$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
+
+all: $(TESTS)
+
+$(TESTS): %: %.o $(LIBTAP_OBJS) $(OBJS_DFOR)
+ $(CC) $(CFLAGS) $(CPPFLAGS) $^ $(LDFLAGS) $(LIBS) -o $@$(X)
+
+check-unit: $(TESTS)
+ echo "# +++ Unit tests in $(subdir) +++" && \
+ cd $(top_builddir)/$(subdir) && \
+ $(PROVE) $(PROVE_FLAGS) \
+ -v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+
+check: check-unit
+
+clean distclean:
+ rm -rf $(TESTS) *.o
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
new file mode 100644
index 00000000000..ce762c52430
--- /dev/null
+++ b/src/test/dfor/meson.build
@@ -0,0 +1,62 @@
+dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
+
+# Object files of vect, bitpack and dfor used by these unit-tests are in the same
+# directory as this meson.build. Tests don't use ones from src/backend/lib and
+# compile different ones for themselves. In Meson/Ninja build system we unite
+# them into a static library.
+
+dfor_sources = files(
+ join_paths(dfor_dir, 'vect_u16.c'),
+ join_paths(dfor_dir, 'bitpack_u16.c'),
+)
+
+dfor_test_lib = static_library(
+ 'dfor_test_lib',
+ dfor_sources,
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ c_args: ['-DFRONTEND'],
+)
+
+# We also build libtap as a static library
+
+libtap = static_library(
+ 'tap',
+ '../../test/libtap/tap.c',
+ include_directories:
+ include_directories('../../..'), # top_builddir
+)
+
+# Each test is an ELF executable
+
+test_names = [
+ 'test_vect_u16',
+ 'test_uniqsortvect_u16',
+ 'test_bitpack_u16',
+]
+
+foreach t : test_names
+ exe = executable(
+ t,
+ t + '.c',
+ link_with: [
+ dfor_test_lib,
+ libtap,
+ common_static, # Provides pg_printf and other common utilities
+ pgport_static, # Provides OS-portability functions
+ ],
+ dependencies: [os_deps, libintl],
+ include_directories: [
+ include_directories('../../..'), # top_builddir
+ include_directories('../'), # src/test
+ postgres_inc, # src/include here
+ ],
+ # Backend code often requires these arguments to identify as backend
+ c_args: ['-DFRONTEND'],
+ )
+
+ test(t, exe, suite: 'dfor')
+endforeach
diff --git a/src/test/dfor/test.h b/src/test/dfor/test.h
new file mode 100644
index 00000000000..f6c54aad95f
--- /dev/null
+++ b/src/test/dfor/test.h
@@ -0,0 +1,31 @@
+
+/*
+ * test.h
+ */
+#ifndef _TEST_H_
+#define _TEST_H_
+
+#include <inttypes.h>
+#include <stdio.h>
+
+static inline void
+test_print_u8_array(size_t cnt, uint8_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%02" PRIx8 ", ", arr[j]);
+
+ printf("%02" PRIx8 " }\n", arr[cnt - 1]);
+}
+
+static inline void
+test_print_u16_array(size_t cnt, uint16_t arr[], const char *name)
+{
+ printf("%s(hex): { ", name);
+ for (size_t j = 0; j < cnt - 1; j++)
+ printf("%04" PRIx16 ", ", arr[j]);
+
+ printf("%04" PRIx16 "}\n", arr[cnt - 1]);
+}
+
+#endif /* _TEST_H_ */
\ No newline at end of file
diff --git a/src/test/dfor/test_bitpack_u16.c b/src/test/dfor/test_bitpack_u16.c
new file mode 100644
index 00000000000..da84bb2f22e
--- /dev/null
+++ b/src/test/dfor/test_bitpack_u16.c
@@ -0,0 +1,357 @@
+/*
+ * test_bitpack.c
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/bitpack_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int
+main(void)
+{
+ plan(195);
+ printf("========================================\n");
+ printf("Test MASK AND WIDTH CALCULATION\n");
+ {
+ cmp_ok(1, "==", (uint16_t)width_u16_from_val(0x00),
+ "Width of 00 is equal to 1 bit.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x02),
+ "Width of 02 is equal to 2 bits.");
+ cmp_ok(2, "==", (uint16_t)width_u16_from_val(0x03),
+ "Width of 03 is equal to 2 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x04),
+ "Width of 04 is equal to 3 bits.");
+ cmp_ok(3, "==", (uint16_t)width_u16_from_val(0x05),
+ "Width of 05 is equal to 3 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x08),
+ "Width of 08 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0A),
+ "Width of 10 is equal to 4 bits.");
+ cmp_ok(4, "==", (uint16_t)width_u16_from_val(0x0F),
+ "Width of 15 is equal to 4 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x10),
+ "Width of 16 is equal to 5 bits.");
+ cmp_ok(5, "==", (uint16_t)width_u16_from_val(0x1F),
+ "Width of 31 is equal to 5 bits.");
+ cmp_ok(6, "==", (uint16_t)width_u16_from_val(0x20),
+ "Width of 32 is equal to 6 bits.");
+ cmp_ok(7, "==", (uint16_t)width_u16_from_val(0x40),
+ "Width of 64 is equal to 7 bits.");
+ cmp_ok(8, "==", (uint16_t)width_u16_from_val(0x80),
+ "Width of 128 is equal to 8 bits.");
+ cmp_ok(13, "==", (uint16_t)width_u16_from_val(0x1000),
+ "Width of 0x01000 is equal to 13 bits.");
+ cmp_ok(16, "==", (uint16_t)width_u16_from_val(0x8ABC),
+ "Width of 0x08ABC is equal to 15 bits.");
+
+ cmp_ok(0x1, "==", (uint16_t)width_u16_to_mask(1),
+ "Mask from width 1 is 00000001(bin).");
+ cmp_ok(0x3, "==", (uint16_t)width_u16_to_mask(2),
+ "Mask from width 2 is 00000011(bin).");
+ cmp_ok(0x7, "==", (uint16_t)width_u16_to_mask(3),
+ "Mask from width 3 is 00000111(bin).");
+ cmp_ok(0x3FF, "==", (uint16_t)width_u16_to_mask(10),
+ "Mask from width 10 is 0x3FF(bin).");
+ cmp_ok(0x7FFF, "==", (uint16_t)width_u16_to_mask(15),
+ "Mask from width 15 is 0x7FFF(bin).");
+ cmp_ok(0xFFFF, "==", (uint16_t)width_u16_to_mask(16),
+ "Mask from width 16 is 0xFFFF(bin).");
+ }
+ printf("Test MASK AND WIDTH CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK PACKING\n");
+ {
+#define SIZEOFPACK 60U
+ uint8_t pack[SIZEOFPACK];
+ size_t caret = 0;
+ memset(pack, 0, SIZEOFPACK);
+ /*
+ * Since we implemented the nulifying of bits according to a mask (see
+ * the bitpack function), we can use even a pack not prepared in advance
+ * and comprising garbage. But we want to check value of each byte of
+ * the pack in this test and we simplify this task by using a zeroed
+ * pack.
+ */
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 1, 1);
+ caret = bitpack_u16_pack(pack, caret, 0, 1);
+ }
+ cmp_ok(16, "==", caret, "Caret = 16.");
+ cmp_ok(0x55, "==", pack[0], "Saved bit-by-bit: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[1], "Saved bit-by-bit: second byte is 0x55.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 10(bin) */, 2);
+ }
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+ cmp_ok(0xAA, "==", pack[2],
+ "Saved with two-bit width: first byte is 0xAA.");
+ cmp_ok(0xAA, "==", pack[3],
+ "Saved with two-bit width: second byte is 0xAA.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x05 /* 101(bin) */, 3);
+ caret = bitpack_u16_pack(pack, caret, 0x02 /* 010(bin) */, 3);
+ }
+
+ cmp_ok(80, "==", caret, "Caret = 80.");
+ cmp_ok(0x55, "==", pack[4],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[5],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[6],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[7],
+ "Saved with three-bit width: first byte is 0x55.");
+ cmp_ok(0x55, "==", pack[8],
+ "Saved with three-bit width: second byte is 0x55.");
+ cmp_ok(0x55, "==", pack[9],
+ "Saved with three-bit width: second byte is 0x55.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x0B, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0C, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0D, 4);
+ caret = bitpack_u16_pack(pack, caret, 0x0E, 4);
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+ cmp_ok(0xCB, "==", pack[10],
+ "Saved with four-bit width: first byte is 0xCB.");
+ cmp_ok(0xED, "==", pack[11],
+ "Saved with four-bit width: second byte is 0xED.");
+
+ for (int j = 0; j < 8; j++) {
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 00111b */, 5);
+ }
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+ cmp_ok(0xE7, "==", pack[12],
+ "Saved with five-bit width: first byte is 0xE7.");
+ cmp_ok(0x9C, "==", pack[13],
+ "Saved with five-bit width: second byte is 0x9C.");
+ cmp_ok(0x73, "==", pack[14],
+ "Saved with five-bit width: third byte is 0x73.");
+ cmp_ok(0xCE, "==", pack[15],
+ "Saved with five-bit width: fourth byte is 0xCE.");
+ cmp_ok(0x39, "==", pack[16],
+ "Saved with five-bit width: fifth byte is 0x39.");
+
+ for (int j = 0; j < 4; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x07 /* 000111b */, 6);
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+ cmp_ok(0xC7, "==", pack[17],
+ "Saved with six-bit width: first byte is 0xC7.");
+ cmp_ok(0x71, "==", pack[18],
+ "Saved with six-bit width: second byte is 0x71.");
+ cmp_ok(0x1C, "==", pack[19],
+ "Saved with six-bit width: third byte is 0x1C.");
+
+ for (int j = 0; j < 8; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x57 /* 1010111b */, 7);
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+ cmp_ok(0xD7, "==", pack[20],
+ "Saved with seven-bit width: byte1 is 0xD7.");
+ cmp_ok(0xEB, "==", pack[21],
+ "Saved with seven-bit width: byte2 is 0xEB.");
+ cmp_ok(0xF5, "==", pack[22],
+ "Saved with seven-bit width: byte3 is 0xF5.");
+ cmp_ok(0x7A, "==", pack[23],
+ "Saved with seven-bit width: byte4 is 0x7A.");
+ cmp_ok(0xBD, "==", pack[24],
+ "Saved with seven-bit width: byte5 is 0xBD.");
+ cmp_ok(0x5E, "==", pack[25],
+ "Saved with seven-bit width: byte6 is 0x5E.");
+ cmp_ok(0xAF, "==", pack[26],
+ "Saved with seven-bit width: byte7 is 0xAF.");
+
+ caret = bitpack_u16_pack(pack, caret, 0xBA, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xDC, 8);
+ caret = bitpack_u16_pack(pack, caret, 0xFE, 8);
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+ cmp_ok(0xBA, "==", pack[27],
+ "Saved with eight-bit width: byte1 is 0xBA.");
+ cmp_ok(0xDC, "==", pack[28],
+ "Saved with eight-bit width: byte2 is 0xDC.");
+ cmp_ok(0xFE, "==", pack[29],
+ "Saved with eight-bit width: byte3 is 0xFE.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ caret = bitpack_u16_pack(pack, caret, 0x32, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x54, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x76, 8);
+ caret = bitpack_u16_pack(pack, caret, 0x98, 8);
+
+ cmp_ok(276, "==", caret, "Caret = 276.");
+ cmp_ok(0x20, "==", pack[30],
+ "Saved with eight-bit width but shifted by 4: is 0x20.");
+ cmp_ok(0x43, "==", pack[31],
+ "Saved with eight-bit width but shifted by 4: is 0x43.");
+ cmp_ok(0x65, "==", pack[32],
+ "Saved with eight-bit width but shifted by 4: is 0x65.");
+ cmp_ok(0x87, "==", pack[33],
+ "Saved with eight-bit width but shifted by 4: is 0x87.");
+ cmp_ok(0x09, "==", pack[34],
+ "Saved with eight-bit width but shifted by 4: is 0x09.");
+
+ caret = bitpack_u16_pack(
+ pack, caret, 0x00,
+ 4); /* move to 4 bit for fun and again continue with 8-bit width */
+ cmp_ok(280, "==", caret, "Caret = 280.");
+ cmp_ok(0x09, "==", pack[34],
+ "Added padding 0x0, width=4. Byte in pack is still 0x09.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x1671 /* 1011001110001b */,
+ 13);
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+ cmp_ok(0x71, "==", pack[35], "Saved with thirteen-bit width: is 0x71.");
+ cmp_ok(0x36, "==", pack[36], "Saved with thirteen-bit width: is 0x36.");
+ cmp_ok(0xCE, "==", pack[37], "Saved with thirteen-bit width: is 0xCE.");
+ cmp_ok(0xC6, "==", pack[38], "Saved with thirteen-bit width: is 0xC6.");
+ cmp_ok(0x59, "==", pack[39], "Saved with thirteen-bit width: is 0x59.");
+
+ caret = bitpack_u16_pack(pack, caret, 0x1 /* PADDING */, 1);
+ cmp_ok(320, "==", caret, "Caret = 320.");
+ cmp_ok(0xD9, "==", pack[39],
+ "After padding with 0x01, w=1: 0x59 -> 0xD9.");
+
+ for (int j = 0; j < 5; j++)
+ caret = bitpack_u16_pack(pack, caret, 0xCDEF, 16);
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ for (int i = 40; i < 50;) {
+ cmp_ok(0xEF, "==", pack[i++], "Packed with width=16. 0xEF.");
+ cmp_ok(0xCD, "==", pack[i++], "Packed with width=16. 0xC.");
+ }
+
+ caret = bitpack_u16_pack(pack, caret,
+ 0x0 /* PADDING in order to shift by 1 bit*/,
+ 1);
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ caret = bitpack_u16_pack(pack, caret, 0x5555, 16);
+
+ cmp_ok(449, "==", caret, "Caret = 401.");
+ for (int i = 50; i < 56;)
+ cmp_ok(0xAA, "==", pack[i++],
+ "16-bit value saved with shift by 1 bit 0x55->0xAA.");
+
+ cmp_ok(0x0, "==", pack[56], "1 higher bit is alone .");
+
+ printf("Test BITPACK PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test BITPACK UNPACKING\n");
+
+ caret = 0;
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x01, "==", bitpack_u16_unpack(pack, &caret, 1));
+ cmp_ok(0x00, "==", bitpack_u16_unpack(pack, &caret, 1));
+ }
+ cmp_ok(caret, "==", 16);
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 2));
+
+ cmp_ok(32, "==", caret, "Caret = 32.");
+
+ for (int j = 0; j < 8; j++) {
+ cmp_ok(0x05, "==", bitpack_u16_unpack(pack, &caret, 3));
+ cmp_ok(0x02, "==", bitpack_u16_unpack(pack, &caret, 3));
+ }
+ cmp_ok(80, "==", caret, "Caret = 80.");
+
+ cmp_ok(0x0B, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0C, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0D, "==", bitpack_u16_unpack(pack, &caret, 4));
+ cmp_ok(0x0E, "==", bitpack_u16_unpack(pack, &caret, 4));
+
+ cmp_ok(96, "==", caret, "Caret = 96.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 5),
+ "width=5, val=00111b");
+
+ cmp_ok(136, "==", caret, "Caret = 136.");
+
+ for (int j = 0; j < 4; j++)
+ cmp_ok(0x07, "==", bitpack_u16_unpack(pack, &caret, 6),
+ "width=6, val=000111b");
+
+ cmp_ok(160, "==", caret, "Caret = 160.");
+
+ for (int j = 0; j < 8; j++)
+ cmp_ok(0x57, "==", bitpack_u16_unpack(pack, &caret, 7),
+ "width=7, val=1010111b (0x57)");
+
+ cmp_ok(216, "==", caret, "Caret = 216.");
+
+ cmp_ok(0xBA, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xBA");
+ cmp_ok(0xDC, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xDC");
+ cmp_ok(0xFE, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0xFE");
+
+ cmp_ok(240, "==", caret, "Caret = 240.");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun");
+
+ cmp_ok(0x32, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x32");
+ cmp_ok(0x54, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x54");
+ cmp_ok(0x76, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x76");
+ cmp_ok(0x98, "==", bitpack_u16_unpack(pack, &caret, 8),
+ "width=8, val=0x98");
+
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 4),
+ "width=4, val=0x0, for fun padding again");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x1671, "==", bitpack_u16_unpack(pack, &caret, 13),
+ "width=13, val=0x1671 (1011001110001b)");
+
+ cmp_ok(319, "==", caret, "Caret = 319.");
+
+ cmp_ok(0x1, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "width=1, val=0x1");
+ cmp_ok(320, "==", caret, "Caret = 320.");
+
+ for (int j = 0; j < 5; j++)
+ cmp_ok(0xCDEF, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "18-bit value alligned with bytes in pack.");
+
+ cmp_ok(400, "==", caret, "Caret = 400.");
+ cmp_ok(0x0, "==", bitpack_u16_unpack(pack, &caret, 1),
+ "1-bit width value (padding for shift).");
+ cmp_ok(401, "==", caret, "Caret = 401.");
+
+ for (int j = 0; j < 3; j++)
+ cmp_ok(0x5555, "==", bitpack_u16_unpack(pack, &caret, 16),
+ "16-bit width value shifted by 1 bit.");
+
+ cmp_ok(449, "==", caret, "Caret = 449.");
+ }
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_uniqsortvect_u16.c b/src/test/dfor/test_uniqsortvect_u16.c
new file mode 100644
index 00000000000..4ddce8b0b3d
--- /dev/null
+++ b/src/test/dfor/test_uniqsortvect_u16.c
@@ -0,0 +1,263 @@
+/*
+ * test_uniqsortvect.c
+ */
+
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+
+int test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in);
+
+int
+test_usv(size_t src_cnt, uint16_t *src_in, size_t expected_cnt,
+ uint16_t *expected_in)
+{
+ int result = -1;
+ vect_u16_t src;
+ vect_u16_t expected;
+ uniqsortvect_u16_t x;
+
+ vect_u16_init(&src, src_cnt, NULL);
+ vect_u16_fill(&src, src_cnt, src_in);
+
+ vect_u16_init(&expected, 0, NULL);
+ vect_u16_fill(&expected, expected_cnt, expected_in);
+
+ vect_u16_init(&x, 0, NULL);
+
+ for (size_t i = 0; i < src_cnt; i++)
+ usv_u16_insert(&x, src.m[i]);
+
+ result = vect_u16_compare(&x, &expected);
+
+ vect_u16_clear(&x);
+ vect_u16_clear(&expected);
+ vect_u16_clear(&src);
+ return result;
+}
+
+int
+main(void)
+{
+ plan(56);
+
+ printf("========================================\n");
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY)\n");
+ {
+ uniqsortvect_u16_t usv;
+ size_t capacity = 10;
+ vect_u16_init(&usv, capacity, NULL);
+ cmp_ok(usv.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(usv.cnt, "==", 0, "No members in vector");
+ ok(usv.m != NULL, "Array for members is reserved");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test CREATE AND DESTROY SORTED UNIQUE VECTOR (EMPTY) PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test SEARCH IN UNIQUE SORT VECT\n");
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[4] = { 5, 10, 20, 30 };
+
+ vect_u16_init(&usv, 0, NULL);
+
+ for (size_t i = 0; i < 4; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 1);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 25);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ srch = usv_u16_search(&usv, 30);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 3, "Pos =2");
+
+ srch = usv_u16_search(&usv, 45);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 3, "Pos =3");
+
+ vect_u16_clear(&usv);
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ uint16_t input[3] = { 5, 10, 20 };
+ uint16_t buf[3]; /* overindulge in testing the outer memory vector */
+
+ vect_u16_init(&usv, 3, buf);
+
+ for (size_t i = 0; i < 3; i++)
+ usv_u16_insert(&usv, input[i]);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 10);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 1, "Pos =1");
+
+ srch = usv_u16_search(&usv, 15);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND, "Not found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ /*
+ * When scopes l and g are neighbours (g-l=1) but
+ * val==m[g] instead of val==m[l].
+ */
+ srch = usv_u16_search(&usv, 20);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ srch = usv_u16_search(&usv, 21);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 2, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* single member*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 3, NULL);
+
+ usv_u16_insert(&usv, 5); /* The only item in list is 5 */
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_SMALLEST,
+ "Not found, smallest");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_FOUND, "Found");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_NOT_FOUND_LARGEST, "Not found, largest");
+ cmp_ok(srch.pos, "==", 0, "Pos =2");
+
+ vect_u16_clear(&usv);
+ }
+ { /* empty vector*/
+
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+
+ vect_u16_init(&usv, 1, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 4);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 5);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ srch = usv_u16_search(&usv, 6);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY, "Vector is empty.");
+ cmp_ok(srch.pos, "==", 0, "Pos =0");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test SEARCH IN UNIQUE SORT VECT PASSED.\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING\n");
+ {
+ usv_srch_res_t srch;
+ srch = usv_u16_search(NULL, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_ERROR,
+ "Error: no vector (empty pointer on vectror).");
+ }
+ {
+ uniqsortvect_u16_t usv;
+ usv_srch_res_t srch;
+ vect_u16_init(&usv, 0, NULL);
+
+ srch = usv_u16_search(&usv, 0);
+ cmp_ok(srch.st, "==", USV_SRCH_EMPTY,
+ "Search in empty vector is not an error.");
+
+ vect_u16_clear(&usv);
+ }
+ printf("Test ERROR ON NO VECTOR FOR SEARCHING PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE\n");
+ {
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unique sorted vector remains the same.");
+
+ cmp_ok(0, "==",
+ test_usv(10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 3, 7, 5, 9 }, // src
+ 8, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 7, 9 }), // expected
+ "Duplicates are removed.");
+
+ cmp_ok(
+ 0, "==",
+ test_usv(10, (uint16_t[]) { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 }, // src
+ 10,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }), // expected
+ "Unsorted became sorted.");
+ }
+ printf("Test CREATED VECTORS ARE SORTED AND UNIQUE PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
diff --git a/src/test/dfor/test_vect_u16.c b/src/test/dfor/test_vect_u16.c
new file mode 100644
index 00000000000..00efe7dccbe
--- /dev/null
+++ b/src/test/dfor/test_vect_u16.c
@@ -0,0 +1,168 @@
+/*
+ * test_vect_u16.c
+ */
+
+#include "libtap/tap.h"
+#include "lib/vect_u16.h"
+
+int
+main(void)
+{
+ plan(35);
+
+ printf("========================================\n");
+ printf("Test INIT AND CLEAR VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ cmp_ok(v.cnt, "==", 0, "No members in vector");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+
+ vect_u16_clear(&v);
+
+ cmp_ok(v.cap, "==", 0, "Vectors capacity is 0 after cleanup");
+ cmp_ok(v.cnt, "==", 0, "No members in vector after cleanup");
+ ok(v.m == NULL, "Array for members is absent");
+ cmp_ok(v.mem_is_outer, "==", false, "Vector does not use outer memory");
+ }
+ printf("Test INIT AND CLEAR VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vectors capacity is equal to requested one");
+ ok(v.m != NULL, "Array for members is reserved");
+ cmp_ok(v.cnt, "==", capacity, "Members are in vector.");
+ {
+ int equal = 0;
+ for (size_t i = 0; i < capacity; i++) {
+ if (v.m[i] == i)
+ equal = equal + 1;
+ else
+ break;
+ }
+ cmp_ok(equal, "==", 10, "Members are correct");
+ }
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test INIT AND FILL VECTOR with zero capcaity\n");
+ {
+ vect_u16_t v;
+ size_t capacity = 0;
+
+ cmp_ok(0, "==", vect_u16_init(&v, capacity, NULL),
+ "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, NULL),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", 0, "Vector's capacity is zero");
+ ok(v.m == NULL,
+ "Pointer to members is NULL (array for members is not reserved)");
+ ok(v.cnt == 0, "Counter of members is zero.");
+ vect_u16_clear(&v);
+ }
+ {
+ vect_u16_t v;
+ size_t capacity = 10;
+ const uint16_t vals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ cmp_ok(0, "==", vect_u16_init(&v, 0, NULL), "Result of vect_init is 0");
+ cmp_ok(0, "==", vect_u16_fill(&v, capacity, vals),
+ "Result of vect_fill is 0");
+
+ cmp_ok(v.cap, "==", capacity,
+ "Vector's capacity is not zero after filling");
+ ok(v.m != NULL,
+ "Pointer to members is not NULL fater filling (array for members has been reserved)");
+ ok(v.cnt == capacity, "Counter of members is not zero after filling.");
+ vect_u16_clear(&v);
+ }
+ printf("Test INIT AND FILL VECTOR with zero capcaity is finished\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test COMPARE VECTORS\n");
+ {
+ vect_u16_t a, b, c, d;
+
+ uint16_t avals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t bvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t cvals[] = { 1, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+ uint16_t dvals[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8 };
+
+ vect_u16_init(&a, 0, NULL);
+ vect_u16_init(&b, 0, NULL);
+ vect_u16_init(&c, 0, NULL);
+ vect_u16_init(&d, 0, NULL);
+
+ vect_u16_fill(&a, 10, avals);
+ vect_u16_fill(&b, 10, bvals);
+ vect_u16_fill(&c, 10, cvals);
+ vect_u16_fill(&d, 9, dvals);
+
+ cmp_ok(0, "==", vect_u16_compare(&a, &b), "Vectors are equal");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &c),
+ "Vectors are not equal because of value of members");
+ cmp_ok(-1, "==", vect_u16_compare(&a, &d),
+ "Vectors are not equal because of number of members");
+ }
+ printf("Test COMPARE VECTORS is finished. \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test VECTOR WITH OUTER MEMORY\n");
+ {
+#define VECT_CAP 100
+ vect_u16_t vect;
+ uint16_t buf[VECT_CAP]; /* uint16_t is the item's type */
+ vect_u16_init(&vect, VECT_CAP, buf);
+ cmp_ok(
+ vect.cap, "==", VECT_CAP,
+ "Initialisation of vector having external memory resulted in proper capacity.");
+ cmp_ok(
+ vect.cnt, "==", 0,
+ "Initialisation of vector having external memory resulted in proper number of items.");
+ ok(((void *)vect.m == (void *)buf),
+ "Initialisation of vector having external memory set buf to vect->m.");
+ ok(vect.mem_is_outer,
+ "Initialisation of vector having external memory set mem_is_outer flag.");
+
+ for (size_t i = 0; i < VECT_CAP; i++)
+ {
+ if (vect_u16_append(&vect, i) != 0)
+ fail(
+ "ERROR: New value can't be appended into vector having external memory.");
+ }
+ pass(
+ "All values have been appended into vector having external memory.");
+
+ cmp_ok(vect.cnt, "==", VECT_CAP, "Vector is full.");
+ cmp_ok(vect_u16_append(&vect, VECT_CAP), "==", -1,
+ "Once vector is full, extra item can't be appended.");
+ }
+ printf("Test VECTOR WITH OUTER MEMORY is finished\n");
+ printf("========================================\n");
+
+ done_testing();
+}
diff --git a/src/test/libtap/.gitignore b/src/test/libtap/.gitignore
new file mode 100644
index 00000000000..2c95d046c7d
--- /dev/null
+++ b/src/test/libtap/.gitignore
@@ -0,0 +1,13 @@
+/t/*
+!/t/*.*
+/t/*.exe
+/t/*.got
+*.a
+*.lo
+*.o
+*.so
+*.pc
+usr/
+*.sw?
+/.deps
+/.dirstamp
diff --git a/src/test/libtap/.travis.yml b/src/test/libtap/.travis.yml
new file mode 100644
index 00000000000..6f9809e1b99
--- /dev/null
+++ b/src/test/libtap/.travis.yml
@@ -0,0 +1,13 @@
+language: c
+
+compiler:
+ - gcc
+ - clang
+
+before_install: sudo apt-get install -y libtest-differences-perl
+
+install: make CC=$CC install
+
+script: make CC=$CC test
+
+after_script: make uninstall
diff --git a/src/test/libtap/COPYING b/src/test/libtap/COPYING
new file mode 100644
index 00000000000..65c5ca88a67
--- /dev/null
+++ b/src/test/libtap/COPYING
@@ -0,0 +1,165 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+ This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+ 0. Additional Definitions.
+
+ As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+ "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+ An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+ A "Combined Work" is a work produced by combining or linking an
+Application with the Library. The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+ The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+ The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+ 1. Exception to Section 3 of the GNU GPL.
+
+ You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+ 2. Conveying Modified Versions.
+
+ If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+ a) under this License, provided that you make a good faith effort to
+ ensure that, in the event an Application does not supply the
+ function or data, the facility still operates, and performs
+ whatever part of its purpose remains meaningful, or
+
+ b) under the GNU GPL, with none of the additional permissions of
+ this License applicable to that copy.
+
+ 3. Object Code Incorporating Material from Library Header Files.
+
+ The object code form of an Application may incorporate material from
+a header file that is part of the Library. You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+ a) Give prominent notice with each copy of the object code that the
+ Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the object code with a copy of the GNU GPL and this license
+ document.
+
+ 4. Combined Works.
+
+ You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+ a) Give prominent notice with each copy of the Combined Work that
+ the Library is used in it and that the Library and its use are
+ covered by this License.
+
+ b) Accompany the Combined Work with a copy of the GNU GPL and this license
+ document.
+
+ c) For a Combined Work that displays copyright notices during
+ execution, include the copyright notice for the Library among
+ these notices, as well as a reference directing the user to the
+ copies of the GNU GPL and this license document.
+
+ d) Do one of the following:
+
+ 0) Convey the Minimal Corresponding Source under the terms of this
+ License, and the Corresponding Application Code in a form
+ suitable for, and under terms that permit, the user to
+ recombine or relink the Application with a modified version of
+ the Linked Version to produce a modified Combined Work, in the
+ manner specified by section 6 of the GNU GPL for conveying
+ Corresponding Source.
+
+ 1) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (a) uses at run time
+ a copy of the Library already present on the user's computer
+ system, and (b) will operate properly with a modified version
+ of the Library that is interface-compatible with the Linked
+ Version.
+
+ e) Provide Installation Information, but only if you would otherwise
+ be required to provide such information under section 6 of the
+ GNU GPL, and only to the extent that such information is
+ necessary to install and execute a modified version of the
+ Combined Work produced by recombining or relinking the
+ Application with a modified version of the Linked Version. (If
+ you use option 4d0, the Installation Information must accompany
+ the Minimal Corresponding Source and Corresponding Application
+ Code. If you use option 4d1, you must provide the Installation
+ Information in the manner specified by section 6 of the GNU GPL
+ for conveying Corresponding Source.)
+
+ 5. Combined Libraries.
+
+ You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+ a) Accompany the combined library with a copy of the same work based
+ on the Library, uncombined with any other library facilities,
+ conveyed under the terms of this License.
+
+ b) Give prominent notice with the combined library that part of it
+ is a work based on the Library, and explaining where to find the
+ accompanying uncombined form of the same work.
+
+ 6. Revised Versions of the GNU Lesser General Public License.
+
+ The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+ If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
diff --git a/src/test/libtap/INSTALL b/src/test/libtap/INSTALL
new file mode 100644
index 00000000000..5b2c76df3d7
--- /dev/null
+++ b/src/test/libtap/INSTALL
@@ -0,0 +1,41 @@
+To install libtap on a Unix-like system:
+
+ $ make
+ $ make check
+ $ make install
+
+To compile with gcc -ansi, run:
+
+ $ ANSI=1 make
+
+To install to a different directory than /usr/local, supply the
+PREFIX variable to make:
+
+ $ PREFIX=/usr make install
+
+On Windows, the library can be created by first setting up the
+correct development environment variables. Usually this is done by
+running vcvars32.bat included in the Visual Studio distribution.
+You should also install gnu make which can be found at
+http://gnuwin32.sourceforge.net/packages/make.htm. Once this is
+done, you should be able to run the following:
+
+ > make -f Makefile.win
+
+If you want to use it directly in another project, you can copy tap.c
+and tap.h there and it shouldn't have a problem compiling.
+
+ $ ls
+ tap.c tap.h test.c
+ $ cat test.c
+ #include "tap.h"
+ int main () {
+ plan(1);
+ ok(50 + 5, "foo %s", "bar");
+ done_testing();
+ }
+ $ gcc test.c tap.c
+ $ a.out
+ 1..1
+ ok 1 - foo bar
+
diff --git a/src/test/libtap/Makefile b/src/test/libtap/Makefile
new file mode 100644
index 00000000000..f020c2839a8
--- /dev/null
+++ b/src/test/libtap/Makefile
@@ -0,0 +1,73 @@
+CC ?= gcc
+CFLAGS += -Wall -I. -fPIC
+PREFIX ?= $(DESTDIR)/usr/local
+TESTS = $(patsubst %.c, %, $(wildcard t/*.c))
+
+ifdef ANSI
+ # -D_BSD_SOURCE for MAP_ANONYMOUS
+ CFLAGS += -ansi -D_BSD_SOURCE
+ LDLIBS += -lbsd-compat
+endif
+
+%:
+ $(CC) $(LDFLAGS) $(TARGET_ARCH) $(filter %.o %.a %.so, $^) $(LDLIBS) -o $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+%.a:
+ $(AR) rcs $@ $(filter %.o, $^)
+
+%.so:
+ $(CC) -shared $(LDFLAGS) $(TARGET_ARCH) $(filter %.o, $^) $(LDLIBS) -o $@
+
+all: libtap.a libtap.so tap.pc tests
+
+tap.pc:
+ @echo generating tap.pc
+ @echo 'prefix='$(PREFIX) > tap.pc
+ @echo 'exec_prefix=$${prefix}' >> tap.pc
+ @echo 'libdir=$${prefix}/lib' >> tap.pc
+ @echo 'includedir=$${prefix}/include' >> tap.pc
+ @echo '' >> tap.pc
+ @echo 'Name: libtap' >> tap.pc
+ @echo 'Description: Write tests in C' >> tap.pc
+ @echo 'Version: 0.1.0' >> tap.pc
+ @echo 'URL: https://github.com/zorgnax/libtap' >> tap.pc
+ @echo 'Libs: -L$${libdir} -ltap' >> tap.pc
+ @echo 'Cflags: -I$${includedir}' >> tap.pc
+
+libtap.a: tap.o
+
+libtap.so: tap.o
+
+tap.o: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %: %.o libtap.a
+
+$(patsubst %, %.o, $(TESTS)): %.o: %.c tap.h
+ $(CC) $(CFLAGS) -O0 $(CPPFLAGS) $(TARGET_ARCH) -c $(filter %.c, $^) $(LDLIBS) -o $@
+
+clean:
+ rm -rf *.o t/*.o tap.pc libtap.a libtap.so $(TESTS)
+
+install: libtap.a tap.h libtap.so tap.pc
+ mkdir -p $(PREFIX)/lib $(PREFIX)/include $(PREFIX)/lib/pkgconfig
+ install -c libtap.a $(PREFIX)/lib
+ install -c libtap.so $(PREFIX)/lib
+ install -c tap.pc $(PREFIX)/lib/pkgconfig
+ install -c tap.h $(PREFIX)/include
+
+uninstall:
+ rm $(PREFIX)/lib/libtap.a $(PREFIX)/lib/libtap.so $(PREFIX)/include/tap.h
+
+dist:
+ rm libtap.zip
+ zip -r libtap *
+
+check test: all
+ ./t/test
+
+.PHONY: all clean install uninstall dist check test tests
diff --git a/src/test/libtap/Makefile.win b/src/test/libtap/Makefile.win
new file mode 100644
index 00000000000..694d679a1b1
--- /dev/null
+++ b/src/test/libtap/Makefile.win
@@ -0,0 +1,37 @@
+CFLAGS = /Zi /Wall /wd4255 /wd4996 /wd4127 /wd4820 /wd4100 /wd4619 \
+ /wd4514 /wd4668 /I.
+CC = cl /nologo
+TESTS = $(patsubst %.c, %.exe, $(wildcard t/*.c))
+
+%.exe:
+ $(CC) $(LDFLAGS) $(filter %.obj %.lib %.dll, $^) $(LDLIBS) /Fe $@
+
+%.o:
+ $(CC) $(CFLAGS) $(CPPFLAGS) /c $(filter %.c, $^) $(LDLIBS) /Fo $@
+
+%.lib:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+%.dll:
+ lib /nologo /out:$@ $(filter %.obj, $^)
+
+all: tap.lib tests
+
+tap.lib: tap.obj
+
+tap.obj: tap.c tap.h
+
+tests: $(TESTS)
+
+$(TESTS): %.exe: %.obj tap.lib
+
+$(patsubst %.exe, %.obj, $(TESTS)): %.obj: %.c tap.h
+
+clean:
+ rm -rf *.obj t/*.obj tap.lib $(TESTS)
+
+check test: all
+ prove
+
+.PHONY: all clean check test tests
+
diff --git a/src/test/libtap/README.md b/src/test/libtap/README.md
new file mode 100644
index 00000000000..5332d526c08
--- /dev/null
+++ b/src/test/libtap/README.md
@@ -0,0 +1,268 @@
+NAME
+====
+
+libtap - Write tests in C
+
+SYNOPSIS
+========
+
+ #include <tap.h>
+
+ int main () {
+ plan(5);
+ int bronze = 1, silver = 2, gold = 3;
+ ok(bronze < silver, "bronze is less than silver");
+ ok(bronze > silver, "not quite");
+ is("gold", "gold", "gold is gold");
+ cmp_ok(silver, "<", gold, "%d <= %d", silver, gold);
+ like("platinum", ".*inum", "platinum matches .*inum");
+ done_testing();
+ }
+
+results in:
+
+ 1..5
+ ok 1 - bronze is less than silver
+ not ok 2 - not quite
+ # Failed test 'not quite'
+ # at t/synopsis.c line 7.
+ ok 3 - gold is gold
+ ok 4 - 2 <= 3
+ ok 5 - platinum matches .*inum
+ # Looks like you failed 1 test of 5 run.
+
+DESCRIPTION
+===========
+
+tap is an easy to read and easy to write way of creating tests for
+your software. This library creates functions that can be used to
+generate it for your C programs. It is implemented using macros
+that include file and line info automatically, and makes it so that
+the format message of each test is optional. It is mostly based on
+the Test::More Perl module.
+
+INSTALL
+=======
+
+On **Unix** systems:
+
+ $ make
+ $ make install
+
+For more detailed installation instructions (eg, for **Windows**), see `INSTALL`.
+
+FUNCTIONS
+=========
+
+- plan(tests)
+- plan(NO_PLAN)
+- plan(SKIP_ALL);
+- plan(SKIP_ALL, fmt, ...)
+
+ Use this to start a series of tests. When you know how many tests there
+ will be, you can put a number as a number of tests you expect to run. If
+ you do not know how many tests there will be, you can use plan(NO_PLAN)
+ or not call this function. When you pass it a number of tests to run, a
+ message similar to the following will appear in the output:
+
+ 1..5
+
+ If you pass it SKIP_ALL, the whole test will be skipped.
+
+- ok(test)
+- ok(test, fmt, ...)
+
+ Specify a test. the test can be any statement returning a true or false
+ value. You may optionally pass a format string describing the test.
+
+ ok(r = reader_new("Of Mice and Men"), "create a new reader");
+ ok(reader_go_to_page(r, 55), "can turn the page");
+ ok(r->page == 55, "page turned to the right one");
+
+ Should print out:
+
+ ok 1 - create a new reader
+ ok 2 - can turn the page
+ ok 3 - page turned to the right one
+
+ On failure, a diagnostic message will be printed out.
+
+ not ok 3 - page turned to the right one
+ # Failed test 'page turned to the right one'
+ # at reader.c line 13.
+
+- is(got, expected)
+- is(got, expected, fmt, ...)
+- isnt(got, unexpected)
+- isnt(got, unexpected, fmt, ...)
+
+ Tests that the string you got is what you expected. with isnt, it is the
+ reverse.
+
+ is("this", "that", "this is that");
+
+ prints:
+
+ not ok 1 - this is that
+ # Failed test 'this is that'
+ # at is.c line 6.
+ # got: 'this'
+ # expected: 'that'
+
+- cmp_ok(a, op, b)
+- cmp_ok(a, op, b, fmt, ...)
+
+ Compares two ints with any binary operator that doesn't require an lvalue.
+ This is nice to use since it provides a better error message than an
+ equivalent ok.
+
+ cmp_ok(420, ">", 666);
+
+ prints:
+
+ not ok 1
+ # Failed test at cmpok.c line 5.
+ # 420
+ # >
+ # 666
+
+- cmp_mem(got, expected, n)
+- cmp_mem(got, expected, n, fmt, ...)
+
+ Tests that the first n bytes of the memory you got is what you expected.
+ NULL pointers for got and expected are handled (if either is NULL,
+ the test fails), but you need to ensure n is not too large.
+
+ char *a = "foo";
+ char *b = "bar";
+ cmp_mem(a, b, 3)
+
+ prints
+
+ not ok 1
+ # Failed test at t/cmp_mem.c line 9.
+ # Difference starts at offset 0
+ # got: 0x66
+ # expected: 0x62
+
+- like(got, expected)
+- like(got, expected, fmt, ...)
+- unlike(got, unexpected)
+- unlike(got, unexpected, fmt, ...)
+
+ Tests that the string you got matches the expected extended POSIX regex.
+ unlike is the reverse. These macros are the equivalent of a skip on
+ Windows.
+
+ like("stranger", "^s.(r).*\\1$", "matches the regex");
+
+ prints:
+
+ ok 1 - matches the regex
+
+- pass()
+- pass(fmt, ...)
+- fail()
+- fail(fmt, ...)
+
+ Speciy that a test succeeded or failed. Use these when the statement is
+ longer than you can fit into the argument given to an ok() test.
+
+- dies_ok(code)
+- dies_ok(code, fmt, ...)
+- lives_ok(code)
+- lives_ok(code, fmt, ...)
+
+ Tests whether the given code causes your program to exit. The code gets
+ passed to a macro that will test it in a forked process. If the code
+ succeeds it will be executed in the parent process. You can test things
+ like passing a function a null pointer and make sure it doesnt
+ dereference it and crash.
+
+ dies_ok({abort();}, "abort does close your program");
+ dies_ok({int x = 0/0;}, "divide by zero crash");
+ lives_ok({pow(3.0, 5.0);}, "nothing wrong with taking 3**5");
+
+ On Windows, these macros are the equivalent of a skip.
+
+- done_testing()
+
+ Summarizes the tests that occurred and exits the main function. If
+ there was no plan, it will print out the number of tests as.
+
+ 1..5
+
+ It will also print a diagnostic message about how many
+ failures there were.
+
+ # Looks like you failed 2 tests of 3 run.
+
+ If all planned tests were successful, it will return 0. If any
+ test fails, it will return 1. If they all passed, but there
+ were missing tests, it will return 2.
+
+- diag(fmt, ...)
+
+ print out a message to the tap output on stdout. Each line is
+ preceeded by a "# " so that you know its a diagnostic message.
+
+ diag("This is\na diag\nto describe\nsomething.");
+
+ prints:
+
+ # This is
+ # a diag
+ # to describe
+ # something
+
+ ok() and this function return an int so you can use it like:
+
+ ok(0) || diag("doh!");
+
+- skip(test, n)
+- skip(test, n, fmt, ...)
+- end_skip
+
+ Skip a series of n tests if test is true. You may give a reason why you are
+ skipping them or not. The (possibly) skipped tests must occur between the
+ skip and end_skip macros.
+
+ skip(TRUE, 2);
+ ok(1);
+ ok(0);
+ end_skip;
+
+ prints:
+
+ ok 1 # skip
+ ok 2 # skip
+
+- todo()
+- todo(fmt, ...)
+- end_todo
+
+ Specifies a series of tests that you expect to fail because they are not
+ yet implemented.
+
+ todo()
+ ok(0);
+ end_todo;
+
+ prints:
+
+ not ok 1 # TODO
+ # Failed (TODO) test at todo.c line 7
+
+- BAIL_OUT()
+- BAIL_OUT(fmt, ...)
+
+ Immediately stops all testing.
+
+ BAIL_OUT("Can't go no further");
+
+ prints
+
+ Bail out! Can't go no further
+
+ and exits with 255.
+
diff --git a/src/test/libtap/tap.c b/src/test/libtap/tap.c
new file mode 100644
index 00000000000..506e4000156
--- /dev/null
+++ b/src/test/libtap/tap.c
@@ -0,0 +1,421 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#define _DEFAULT_SOURCE 1
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "tap.h"
+
+#ifndef _WIN32
+#include <sys/types.h>
+#include <sys/param.h>
+#include <sys/mman.h>
+
+#include <regex.h>
+
+#ifndef MAP_ANONYMOUS
+#ifdef MAP_ANON
+#define MAP_ANONYMOUS MAP_ANON
+#else
+#error "System does not support mapping anonymous pages"
+#endif
+#endif
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define PRINTF_ATTR(fmtarg, firstvararg) __attribute__((format(printf, fmtarg, firstvararg)))
+#else
+#define PRINTF_ATTR(fmtarg, firstvararg)
+#endif
+
+static int expected_tests = NO_PLAN;
+static int failed_tests;
+static int current_test;
+static char *todo_mesg;
+
+static char *vstrdupf(const char *fmt, va_list args) PRINTF_ATTR(1,0);
+
+void tap_plan(int tests, const char *fmt, ...) PRINTF_ATTR(2, 3);
+
+int vok_at_loc(const char *file, int line, int test, const char *fmt,
+ va_list args) PRINTF_ATTR(4,0);
+
+int ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+ PRINTF_ATTR(4, 5);
+
+int is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...) PRINTF_ATTR(5, 6);
+
+int isnt_at_loc(const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(5, 6);
+
+int cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+int diag(const char *fmt, ...) PRINTF_ATTR(1, 2);
+
+int bail_out(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_skip(int n, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+void tap_todo(int ignore, const char *fmt, ...)
+ PRINTF_ATTR(2, 3);
+
+int like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+ PRINTF_ATTR(6, 7);
+
+static char *
+vstrdupf(const char *fmt, va_list args)
+{
+ char *str;
+ int size;
+ va_list args2;
+ va_copy(args2, args);
+ if (!fmt)
+ fmt = "";
+ size = vsnprintf(NULL, 0, fmt, args2) + 2;
+ str = malloc(size);
+ if (!str) {
+ perror("malloc error");
+ exit(1);
+ }
+ vsprintf(str, fmt, args);
+ va_end(args2);
+ return str;
+}
+
+void
+tap_plan(int tests, const char *fmt, ...)
+{
+ expected_tests = tests;
+ if (tests == SKIP_ALL) {
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ printf("1..0 ");
+ diag("SKIP %s\n", why);
+ exit(0);
+ }
+ if (tests != NO_PLAN) {
+ printf("1..%d\n", tests);
+ }
+}
+
+int
+vok_at_loc(const char *file, int line, int test, const char *fmt, va_list args)
+{
+ char *name = vstrdupf(fmt, args);
+ if (!test) {
+ printf("not ");
+ }
+ printf("ok %d", ++current_test);
+ if (*name)
+ printf(" - %s", name);
+ if (todo_mesg) {
+ printf(" # TODO");
+ if (*todo_mesg)
+ printf(" %s", todo_mesg);
+ }
+ printf("\n");
+ if (!test) {
+ printf("# Failed ");
+ if (todo_mesg)
+ printf("(TODO) ");
+ printf("test ");
+ if (*name)
+ printf("'%s'\n# ", name);
+ printf("at %s line %d.\n", file, line);
+ if (!todo_mesg)
+ failed_tests++;
+ }
+ free(name);
+ return test;
+}
+
+int
+ok_at_loc(const char *file, int line, int test, const char *fmt, ...)
+{
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ return test;
+}
+
+static int
+mystrcmp(const char *a, const char *b)
+{
+ return a == b ? 0 : !a ? -1 : !b ? 1 : strcmp(a, b);
+}
+
+#define eq(a, b) (!mystrcmp(a, b))
+#define ne(a, b) (mystrcmp(a, b))
+
+int
+is_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = eq(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: '%s'", expected);
+ }
+ return test;
+}
+
+int
+isnt_at_loc(const char *file, int line, const char *got, const char *expected,
+ const char *fmt, ...)
+{
+ int test = ne(got, expected);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" got: '%s'", got);
+ diag(" expected: anything else");
+ }
+ return test;
+}
+
+int
+cmp_ok_at_loc(const char *file, int line, int a, const char *op, int b,
+ const char *fmt, ...)
+{
+ int test = eq(op, "||") ? a || b :
+ eq(op, "&&") ? a && b :
+ eq(op, "|") ? a | b :
+ eq(op, "^") ? a ^ b :
+ eq(op, "&") ? a & b :
+ eq(op, "==") ? a == b :
+ eq(op, "!=") ? a != b :
+ eq(op, "<") ? a < b :
+ eq(op, ">") ? a > b :
+ eq(op, "<=") ? a <= b :
+ eq(op, ">=") ? a >= b :
+ eq(op, "<<") ? a << b :
+ eq(op, ">>") ? a >> b :
+ eq(op, "+") ? a + b :
+ eq(op, "-") ? a - b :
+ eq(op, "*") ? a * b :
+ eq(op, "/") ? a / b :
+ eq(op, "%") ? a % b :
+ diag("unrecognized operator '%s'", op);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ diag(" %d", a);
+ diag(" %s", op);
+ diag(" %d", b);
+ }
+ return test;
+}
+
+static int
+find_mem_diff(const char *a, const char *b, size_t n, size_t *offset)
+{
+ size_t i;
+ if (a == b)
+ return 0;
+ if (!a || !b)
+ return 2;
+ for (i = 0; i < n; i++) {
+ if (a[i] != b[i]) {
+ *offset = i;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int
+cmp_mem_at_loc(const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...)
+{
+ size_t offset;
+ int diff = find_mem_diff(got, expected, n, &offset);
+ va_list args;
+ va_start(args, fmt);
+ vok_at_loc(file, line, !diff, fmt, args);
+ va_end(args);
+ if (diff == 1) {
+ diag(" Difference starts at offset %lu", offset);
+ diag(" got: 0x%02x", ((const unsigned char *)got)[offset]);
+ diag(" expected: 0x%02x", ((const unsigned char *)expected)[offset]);
+ } else if (diff == 2) {
+ diag(" got: %s", got ? "not NULL" : "NULL");
+ diag(" expected: %s", expected ? "not NULL" : "NULL");
+ }
+ return !diff;
+}
+
+int
+diag(const char *fmt, ...)
+{
+ va_list args;
+ char *mesg, *line;
+ int i;
+ va_start(args, fmt);
+ if (!fmt) {
+ va_end(args);
+ return 0;
+ }
+ mesg = vstrdupf(fmt, args);
+ line = mesg;
+ for (i = 0; *line; i++) {
+ char c = mesg[i];
+ if (!c || c == '\n') {
+ mesg[i] = '\0';
+ printf("# %s\n", line);
+ if (!c)
+ break;
+ mesg[i] = c;
+ line = mesg + i + 1;
+ }
+ }
+ free(mesg);
+ va_end(args);
+ return 0;
+}
+
+int
+exit_status(void)
+{
+ int retval = 0;
+ if (expected_tests == NO_PLAN) {
+ printf("1..%d\n", current_test);
+ } else if (current_test != expected_tests) {
+ diag("Looks like you planned %d test%s but ran %d.", expected_tests,
+ expected_tests > 1 ? "s" : "", current_test);
+ retval = 2;
+ }
+ if (failed_tests) {
+ diag("Looks like you failed %d test%s of %d run.", failed_tests,
+ failed_tests > 1 ? "s" : "", current_test);
+ retval = 1;
+ }
+ return retval;
+}
+
+int
+bail_out(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ printf("Bail out! ");
+ vprintf(fmt, args);
+ printf("\n");
+ va_end(args);
+ exit(255);
+ return 0;
+}
+
+void
+tap_skip(int n, const char *fmt, ...)
+{
+ char *why;
+ va_list args;
+ va_start(args, fmt);
+ why = vstrdupf(fmt, args);
+ va_end(args);
+ while (n-- > 0) {
+ printf("ok %d ", ++current_test);
+ diag("skip %s\n", why);
+ }
+ free(why);
+}
+
+void
+tap_todo(int ignore, const char *fmt, ...)
+{
+ va_list args;
+ (void)ignore;
+ va_start(args, fmt);
+ todo_mesg = vstrdupf(fmt, args);
+ va_end(args);
+}
+
+void
+tap_end_todo(void)
+{
+ free(todo_mesg);
+ todo_mesg = NULL;
+}
+
+#ifndef _WIN32
+/* Create a shared memory int to keep track of whether a piece of code executed
+dies. to be used in the dies_ok and lives_ok macros. */
+int
+tap_test_died(int status)
+{
+ static int *test_died = NULL;
+ int prev;
+ if (!test_died) {
+ test_died = mmap(0, sizeof(int), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ *test_died = 0;
+ }
+ prev = *test_died;
+ *test_died = status;
+ return prev;
+}
+
+int
+like_at_loc(int for_match, const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...)
+{
+ int test;
+ regex_t re;
+ va_list args;
+ int err = regcomp(&re, expected, REG_EXTENDED);
+ if (err) {
+ char errbuf[256];
+ regerror(err, &re, errbuf, sizeof errbuf);
+ fprintf(stderr, "Unable to compile regex '%s': %s at %s line %d\n",
+ expected, errbuf, file, line);
+ exit(255);
+ }
+ err = regexec(&re, got, 0, NULL, 0);
+ regfree(&re);
+ test = for_match ? !err : err;
+ va_start(args, fmt);
+ vok_at_loc(file, line, test, fmt, args);
+ va_end(args);
+ if (!test) {
+ if (for_match) {
+ diag(" '%s'", got);
+ diag(" doesn't match: '%s'", expected);
+ } else {
+ diag(" '%s'", got);
+ diag(" matches: '%s'", expected);
+ }
+ }
+ return test;
+}
+#endif
diff --git a/src/test/libtap/tap.h b/src/test/libtap/tap.h
new file mode 100644
index 00000000000..e366a6affdc
--- /dev/null
+++ b/src/test/libtap/tap.h
@@ -0,0 +1,115 @@
+/*
+libtap - Write tests in C
+Copyright 2012 Jake Gelbman <[email protected]>
+This file is licensed under the LGPL
+*/
+
+#ifndef __TAP_H__
+#define __TAP_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef va_copy
+#ifdef __va_copy
+#define va_copy __va_copy
+#else
+#define va_copy(d, s) ((d) = (s))
+#endif
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+
+int vok_at_loc (const char *file, int line, int test, const char *fmt,
+ va_list args);
+int ok_at_loc (const char *file, int line, int test, const char *fmt,
+ ...);
+int is_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int isnt_at_loc (const char *file, int line, const char *got,
+ const char *expected, const char *fmt, ...);
+int cmp_ok_at_loc (const char *file, int line, int a, const char *op,
+ int b, const char *fmt, ...);
+int cmp_mem_at_loc (const char *file, int line, const void *got,
+ const void *expected, size_t n, const char *fmt, ...);
+int bail_out (int ignore, const char *fmt, ...);
+void tap_plan (int tests, const char *fmt, ...);
+int diag (const char *fmt, ...);
+int exit_status (void);
+void tap_skip (int n, const char *fmt, ...);
+void tap_todo (int ignore, const char *fmt, ...);
+void tap_end_todo (void);
+
+#define NO_PLAN -1
+#define SKIP_ALL -2
+#define ok(...) ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define is(...) is_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define isnt(...) isnt_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_ok(...) cmp_ok_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define cmp_mem(...) cmp_mem_at_loc(__FILE__, __LINE__, __VA_ARGS__, NULL)
+#define plan(...) tap_plan(__VA_ARGS__, NULL)
+#define done_testing() return exit_status()
+#define BAIL_OUT(...) bail_out(0, "" __VA_ARGS__, NULL)
+#define pass(...) ok(1, "" __VA_ARGS__)
+#define fail(...) ok(0, "" __VA_ARGS__)
+
+#define skip(test, ...) do {if (test) {tap_skip(__VA_ARGS__, NULL); break;}
+#define end_skip } while (0)
+
+#define todo(...) tap_todo(0, "" __VA_ARGS__, NULL)
+#define end_todo tap_end_todo()
+
+#define dies_ok(...) dies_ok_common(1, __VA_ARGS__)
+#define lives_ok(...) dies_ok_common(0, __VA_ARGS__)
+
+#ifdef _WIN32
+#define like(...) tap_skip(1, "like is not implemented on Windows")
+#define unlike(...) tap_skip(1, "unlike is not implemented on Windows")
+#define dies_ok_common(...) \
+ tap_skip(1, "Death detection is not supported on Windows")
+#else
+#define like(...) like_at_loc(1, __FILE__, __LINE__, __VA_ARGS__, NULL)
+#define unlike(...) like_at_loc(0, __FILE__, __LINE__, __VA_ARGS__, NULL)
+int like_at_loc (int for_match, const char *file, int line,
+ const char *got, const char *expected,
+ const char *fmt, ...);
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+int tap_test_died (int status);
+#define dies_ok_common(for_death, code, ...) \
+ do { \
+ int cpid; \
+ int it_died; \
+ tap_test_died(1); \
+ cpid = fork(); \
+ switch (cpid) { \
+ case -1: \
+ perror("fork error"); \
+ exit(1); \
+ case 0: \
+ close(1); \
+ close(2); \
+ code \
+ tap_test_died(0); \
+ exit(0); \
+ } \
+ if (waitpid(cpid, NULL, 0) < 0) { \
+ perror("waitpid error"); \
+ exit(1); \
+ } \
+ it_died = tap_test_died(0); \
+ if (!it_died) \
+ {code} \
+ ok(for_death ? it_died : !it_died, "" __VA_ARGS__); \
+ } while (0)
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/src/test/meson.build b/src/test/meson.build
index cd45cbf57fb..64fa751a5a5 100644
--- a/src/test/meson.build
+++ b/src/test/meson.build
@@ -8,6 +8,7 @@ subdir('postmaster')
subdir('recovery')
subdir('subscription')
subdir('modules')
+subdir('dfor')
if ssl.found()
subdir('ssl')
--
2.53.0
[text/x-patch] v13-0003-Implement-Delta-Frame-of-Reference-compression.patch (31.8K, 4-v13-0003-Implement-Delta-Frame-of-Reference-compression.patch)
download | inline diff:
From 28a50c25da511db0a59f32430aac1d62d0379177 Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:15:34 +0800
Subject: [PATCH v13 3/5] Implement Delta Frame of Reference compression.
Implement the compression algorithm based on the Delta Frame of
Reference technique (DFOR).
The current implementation of the DFOR algorithm supports items of
unsigned integer types. The type of an item is defined with the
item_t macro.
The compressed pack comprises three parts:
- deltas;
- exceptions positions.
- exceptions;
Each of them is a densily bit-packed sequence.
The delta is a difference between the current item and the previous one. The
delta of the first item (the item having the zero index) is its actual value:
delta[0] = m[0]-0 = m[0]. Serialised deltas is a sequence of bits. Each
serialised delta in 'deltas' has a fixed bit width. If the delta's width
exceeds the allowed size of a delta in 'deltas', the higher bits of this
delta is put into exceptions, its positions is saved into the 'excepton
positions' section. Usage of exceptions can be deliberately turned off.
DFoR supports both external memory (outer memory) provided by a caller
and automatically managed memory, allocated by means of malloc, palloc
or similar functions. Memory management configuration must be defined
during initialization. All subsequent operations follow this
configuration. For example, a caller can place a buffer on the stack to
avoid heap allocation and pass the buffer to a DFoR unit. As a result,
the packing and unpacking processes exclude dynamic allocation.
The DFoR unit is implemented as a set of templates. Developers can
generate DFoR implementations for any unsigned integer type (uint8_t,
uint16_t, uint32_t, uint64_t). The dfor_u16 unit is implemented.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/lib/Makefile.dfor | 1 +
src/backend/lib/dfor_templ.c | 722 ++++++++++++++++++++++++++++
src/backend/lib/dfor_u16.c | 8 +
src/backend/lib/meson.build | 1 +
src/include/lib/dfor_templ.h | 27 ++
src/include/lib/dfor_templ_staple.h | 126 +++++
src/include/lib/dfor_templ_undef.h | 28 ++
src/include/lib/dfor_u16.h | 13 +
src/include/lib/dfor_u16_config.h | 4 +
src/tools/pginclude/headerscheck | 4 +
10 files changed, 934 insertions(+)
create mode 100644 src/backend/lib/dfor_templ.c
create mode 100644 src/backend/lib/dfor_u16.c
create mode 100644 src/include/lib/dfor_templ.h
create mode 100644 src/include/lib/dfor_templ_staple.h
create mode 100644 src/include/lib/dfor_templ_undef.h
create mode 100644 src/include/lib/dfor_u16.h
create mode 100644 src/include/lib/dfor_u16_config.h
diff --git a/src/backend/lib/Makefile.dfor b/src/backend/lib/Makefile.dfor
index b93c6e78644..beb7035f155 100644
--- a/src/backend/lib/Makefile.dfor
+++ b/src/backend/lib/Makefile.dfor
@@ -2,4 +2,5 @@
OBJS_DFOR := \
bitpack_u16.o \
+ dfor_u16.o \
vect_u16.o
diff --git a/src/backend/lib/dfor_templ.c b/src/backend/lib/dfor_templ.c
new file mode 100644
index 00000000000..48280b2b9c1
--- /dev/null
+++ b/src/backend/lib/dfor_templ.c
@@ -0,0 +1,722 @@
+/*
+ * dfor_templ.c
+ *
+ * Implement the variant of the Frame of Reference with Delta
+ * container and corresponding algorithms.
+ *
+ * The current implementation of DFOR algorithm supports items of unsigned
+ * integer type. The type of original items is defined with the item_t macro.
+ *
+ * The compressed pack comprises three parts:
+ * - deltas (least significant bits of deltas);
+ * - exceptions positions.
+ * - exceptions (most significant bits of deltas);
+ * Each of them is a densily bit-packed sequence.
+ *
+ * The delta is a difference between the current item and the previous one. The
+ * delta of the first item (the item having the zero index) is its actual value:
+ * delta[0] = m[0]-0 = m[0]. A serialised delta is a sequence of bits. Each
+ * serialised delta in 'deltas' has a fixed bit width. If the delta's width
+ * exceeds some critical size, defined during analyzing, the higher bits of this
+ * delta is put into exceptions, which position is saved into the 'excepton
+ * positions' section. Usage of exceptions can be deliberately turned off.
+ *
+ * If signed deltas are used, the module of a delta is represented by most
+ * siginificant bits and the sign is saved as the least significant bit: 0 -
+ * delta is positive, 1 - delta is negative.
+ *
+ * DFoR supports both external memory (outer memory) provided by a caller
+ * and automatically managed memory, allocated by means of malloc, palloc
+ * or similar functions. Memory management configuration must be defined
+ * during initialization. All subsequent operations follow this
+ * configuration. For example, a caller can place a buffer on the stack to
+ * avoid heap allocation and pass the buffer to a DFoR unit. As a result,
+ * the packing and unpacking processes exclude dynamic allocation.
+ *
+ * The DFoR unit is implemented as a set of templates. Developers can
+ * generate DFoR implementations for any unsigned integer type (uint8_t,
+ * uint16_t, uint32_t, uint64_t).
+ */
+
+#include "lib/dfor_templ_staple.h"
+
+int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters,
+ bool *sign);
+
+int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+int dfor_analyze(size_t cnt, const item_t arr[],
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos);
+
+int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[]);
+
+void dfor_clear_meta(dfor_meta_t *dfor);
+
+dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+/*
+ * Calculate deltas
+ *
+ * vWidthCounters being equal to NULL means 'Do not calculate counts of widths'.
+ * In this case usvDeltaWidth comprise only one member m[0] which saves max
+ * width of delta, which can be used by caller.
+ *
+ * The value of the signed_deltas argument can be changed by this function from
+ * false to true, but not vice versa. Once signed_deltas is set to true, it cannot
+ * be changed.
+ */
+int
+dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths, vect_t *vWidthCounters,
+ bool *signedDeltas)
+{
+ item_t delta;
+ item_t prev; /* value of previous number*/
+ size_t width;
+ size_t maxItemWidth;
+ item_t maxItemMask;
+ item_t maxModulusOfDelta;
+
+ usv_ins_res_t insWidthInsert;
+
+ if (vDeltas == NULL || usvDeltaWidths == NULL || signedDeltas == NULL)
+ return -1;
+
+ vect_clear(vDeltas);
+ vect_clear(usvDeltaWidths);
+
+ if (vWidthCounters == NULL)
+ usv_insert(usvDeltaWidths, 0);
+ else
+ vect_clear(vWidthCounters);
+
+ /*
+ * We use the maximum possible item width here which is equal ro the width
+ * of item_t. TODO: we can analyze items and calculate the real max item
+ * width. This must improve compression ratio.
+ */
+ maxItemWidth = sizeof(item_t) * 8;
+ maxItemMask = width_to_mask(maxItemWidth);
+ maxModulusOfDelta = (*signedDeltas) ? maxItemMask >> 1 : maxItemMask;
+
+ prev = 0;
+ for (size_t j = 0; j < cnt; j++)
+ {
+ bool negDelta = arr[j] < prev;
+
+ if (negDelta)
+ {
+ if (unlikely(*signedDeltas == false))
+ { /* Recalculate all deltas as signed. */
+ *signedDeltas = true;
+ return dfor_calc_deltas(cnt, arr, vDeltas, usvDeltaWidths,
+ vWidthCounters, signedDeltas);
+ }
+ Assert(*signedDeltas == true);
+ delta = prev - arr[j];
+ }
+ else
+ delta = arr[j] - prev;
+
+ if (delta > maxModulusOfDelta)
+ {
+ Assert(*signedDeltas == true);
+ /* Use trick with overlapping here */
+ delta = maxItemMask - delta + 1;
+ negDelta = !negDelta;
+ }
+
+ Assert(delta <= maxModulusOfDelta);
+
+ if (*signedDeltas == true)
+ {
+ uint8 signCode = (negDelta) ? 1 : 0;
+ delta = delta << 1 | signCode;
+ width = (delta == 0) ? 2 : width_from_val(delta);
+ }
+ else
+ width = width_from_val(delta);
+
+ vect_append(vDeltas, delta);
+
+ if (vWidthCounters == NULL)
+ {
+ if (usvDeltaWidths->m[0] < width)
+ usvDeltaWidths->m[0] = width;
+ }
+ else
+ {
+ insWidthInsert = usv_insert(usvDeltaWidths, width);
+
+ if (insWidthInsert.st == USV_INS_NEW)
+ vect_insert(vWidthCounters, insWidthInsert.pos, (item_t)1);
+ else if (insWidthInsert.st == USV_INS_EXISTS)
+ vWidthCounters->m[insWidthInsert.pos]++;
+ else
+ return -1;
+ }
+ prev = arr[j];
+ }
+ return 0;
+}
+
+/*
+ * Calculate width of short deltas, width of exceptions, and number of
+ * exceptions
+ */
+int
+dfor_calc_width(size_t cntDelta, const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions)
+{
+#define MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS 4
+
+ size_t cntShortDeltas; /* number of deltas presented without exceptions */
+ size_t indxWidth; /* the width of short deltas (index from vWidthCounters
+ * (and from vDeltaWidth accordingly)
+ */
+ if (usvDeltaWidths == NULL || vWidthCounters == NULL || width == NULL ||
+ cntExceptions == NULL)
+ return -1;
+
+ cntShortDeltas = cntDelta;
+ indxWidth = usvDeltaWidths->cnt - 1; /* counter into index */
+ *cntExceptions = 0;
+
+ /*
+ * Here we try to decrease the width of short deltas in order to compress
+ * the array of deltas in the meantime we are eager to cover no less than
+ * 90% of deltas we have. It is an heuristic analysis based on the
+ * suggestion "no less than 90% of deltas we have".
+ *
+ * TODO: analyzing we might want calulate the full size of the pack for each
+ * variant of the width.
+ */
+ if (cntDelta >= MIN_DELTAS_QUANTITY_ALLOWING_EXCEPTIONS) {
+ size_t szMinCoverage; /* threshold */
+ size_t j;
+
+ if (cntDelta >= 10)
+ szMinCoverage = cntDelta - cntDelta / 10;
+ else
+ szMinCoverage = cntDelta - 1;
+
+ j = indxWidth;
+
+ while (j > 0) {
+ if (cntShortDeltas - vWidthCounters->m[j] < szMinCoverage)
+ break;
+
+ cntShortDeltas -= vWidthCounters->m[j];
+ j--;
+ indxWidth = j;
+ }
+ *cntExceptions = cntDelta - cntShortDeltas;
+ }
+
+ *width = usvDeltaWidths->m[indxWidth];
+ return 0;
+}
+
+/*
+ * dfor_analyze
+ * Analyze input array, calculate deltas and their width, define exceptions and
+ * their positions. Returns them through the dfor, vDeltas, usvExcPos. If
+ * usvExcPos == NULL - don't calculate exceptions.
+ *
+ * dfor_analyze function does not use dynamic memory allocation for its
+ * local containers.
+ *
+ * A caller has to control whether vDeltas and usvExcPos use outer memory
+ * provided by caller or manage memory allocation automatically, which defines
+ * whether vect_insert and vect_append functions, invoked from here, use dynamic
+ * memory or not.
+ *
+ * A caller should take into account that dfor_meta_t dfor are going to be
+ * nullified in this function, so it should not have any meaningfull data by
+ * start of dfor_analyze, especially its pack field should not be used as a
+ * pointer on dynamic memory, otherwise memory leakage is possible.
+ *
+ */
+int
+dfor_analyze(size_t cnt, const item_t arr[], /* input */
+ dfor_meta_t *dfor, vect_t *vDeltas,
+ uniqsortvect_t *usvExcPos) /* output */
+{
+#define DELTA_WIDTH_MAX_NUMBER (sizeof(item_t) * 8)
+ uniqsortvect_t usvDeltaWidths;
+ item_t bufDeltaWidth[DELTA_WIDTH_MAX_NUMBER];
+ vect_t vWidthCounters;
+ item_t bufWidthCounters[DELTA_WIDTH_MAX_NUMBER];
+ item_t mask;
+ int res = -1;
+
+ excalg_t isExcUsage = (usvExcPos == NULL) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (dfor == NULL)
+ goto dfor_analyze_error;
+
+ memset(dfor, 0, sizeof(dfor_meta_t));
+
+ if (cnt == 0)
+ /* dfor->item_cnt = 0; */ /* it's been already done with memset */
+ goto dfor_analyze_ret;
+ else if (arr == NULL)
+ goto dfor_analyze_error;
+
+ if (0 != vect_init(&usvDeltaWidths, DELTA_WIDTH_MAX_NUMBER, bufDeltaWidth))
+ goto dfor_analyze_error;
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ vect_init(&vWidthCounters, DELTA_WIDTH_MAX_NUMBER,
+ bufWidthCounters))
+ goto dfor_analyze_error;
+ }
+
+ dfor->item_cnt = cnt;
+ dfor->signed_deltas = false;
+ if (0 !=
+ dfor_calc_deltas(dfor->item_cnt, arr, vDeltas, &usvDeltaWidths,
+ (isExcUsage == DFOR_EXC_USE) ? &vWidthCounters : NULL,
+ &dfor->signed_deltas))
+ goto dfor_analyze_error;
+
+ Assert(cnt == vDeltas->cnt);
+ Assert(usvDeltaWidths.cnt > 0);
+
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ if (0 !=
+ dfor_calc_width(vDeltas->cnt, &usvDeltaWidths, &vWidthCounters,
+ &dfor->delta_wid, &dfor->exc_cnt))
+ goto dfor_analyze_error;
+ }
+ else
+ {
+ dfor->delta_wid =
+ usvDeltaWidths.m[usvDeltaWidths.cnt - 1]; /* max width */
+ dfor->exc_cnt = 0;
+ }
+
+ dfor->exc_wid = usvDeltaWidths.m[usvDeltaWidths.cnt - 1] - dfor->delta_wid;
+
+ /* A mask looks like 0001111. It is also the max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ for (size_t i = 0; i < vDeltas->cnt; i++)
+ {
+ if (vDeltas->m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ if (0 != vect_append(usvExcPos, (item_t)i))
+ goto dfor_analyze_error;
+ }
+ }
+ Assert(dfor->delta_wid + dfor->exc_wid <= sizeof(item_t) * 8);
+ res = 0;
+dfor_analyze_ret:
+ return res;
+dfor_analyze_error:
+ /* dfor_analyze doesn't affect the pack field (doesn't allocate, delete or
+ * otherwise), so we can nullify the whole dfor and it
+ * is safe, no leakage */
+ memset(dfor, 0, sizeof(dfor_meta_t));
+ res = -1;
+ goto dfor_analyze_ret;
+}
+
+/*
+ * dfor_pack
+ *
+ * The input array arr has to be sorted.
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to
+ * provide the external memory buffer. The size of this buffer should be not
+ * less than 4 * cnt * sizeof(item_t). It will be used for arrays pointed by
+ * *(dfor->pack), *(vDeltas->m), *(vExcPosDeltas->m), *(usvExcPos->m).
+ *
+ * If dynamic allocation has been used by the dfor_pack, a caller has to free
+ * the piece of memory pointed by dfor->pack, since it is alocated by the
+ * dfor_pack with DFOR_MALLOC. Freeing has to be performed by function
+ * conforming to DFOR_MALLOC (paired with it). For instance, if DFOR_MALLOC is
+ * malloc, than memory should be freed by free.
+ */
+int
+dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[])
+{
+ int res;
+ vect_t vDeltas = { 0 };
+ vect_t vExcPosDeltas = { 0 };
+ uniqsortvect_t usvExcPos = { 0 };
+
+ if (dfor == NULL ||
+ (bufSize != 0 && bufSize < 4 * cnt * sizeof(item_t)))
+ {
+ goto dfor_pack_error;
+ }
+
+ /*
+ * We don't need it here:
+ * memset(dfor, 0, sizeof(dfor_meta_t)).
+ * It is going to be done in dfor_analyze.
+ */
+
+ {
+ item_t *deltaBuf = NULL;
+ item_t *excPosDeltasBuf = NULL;
+ item_t *excPosBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+ int res3 = 0;
+
+ if (bufSize != 0)
+ {
+ /* Step over the maximal allowed DFoR pack size */
+ deltaBuf = (item_t*)(buf + cnt * sizeof(item_t));
+ excPosDeltasBuf = (item_t*)(buf + cnt * sizeof(item_t) * 2);
+ excPosBuf = (item_t*)(buf + cnt * sizeof(item_t) * 3);
+ }
+
+ /* Setup containers with outer memory */
+ res1 = vect_init(&vDeltas, cnt, deltaBuf);
+
+ if (isExcUsage)
+ {
+ res2 = vect_init(&vExcPosDeltas, cnt, excPosDeltasBuf);
+ res3 = vect_init(&usvExcPos, cnt, excPosBuf);
+ }
+
+ if (res1 != 0 || res2 != 0 || res3 != 0)
+ goto dfor_pack_error;
+ }
+
+ if (0 !=
+ dfor_analyze(cnt, arr, dfor, &vDeltas,
+ (isExcUsage == DFOR_EXC_USE) ? &usvExcPos : NULL))
+ goto dfor_pack_error;
+
+ if (dfor->exc_cnt != 0)
+ {
+ /* We treat exception positions as a sorted sequence, apply the
+ * DFoR algorithm to it, and save not their absolute values but their
+ * deltas. */
+ dfor_meta_t dforExcPos;
+ Assert(dfor->exc_cnt == usvExcPos.cnt);
+ if (0 !=
+ dfor_analyze(usvExcPos.cnt, usvExcPos.m, &dforExcPos,
+ &vExcPosDeltas, NULL))
+ goto dfor_pack_error;
+
+ Assert(dfor->exc_cnt == vExcPosDeltas.cnt);
+ Assert(dfor->exc_cnt == dforExcPos.item_cnt);
+
+ dfor->exc_pos_wid = dforExcPos.delta_wid;
+ }
+ else
+ {
+ Assert(usvExcPos.cnt == 0); /* usvExcPos has to remain zeroed. */
+ Assert(dfor->exc_wid == 0); /* No exceptions, no exceptions' width. */
+ Assert(dfor->exc_pos_wid == 0); /* No exceptions' positions width too. */
+ }
+
+ /* dfor_pack serialisation packing */
+ {
+ /* index of the next free bit to be used: */
+ size_t d; /* - by a delta */
+ size_t e; /* - by an exception */
+ size_t p; /* - by an exception position */
+ item_t mask;
+ dfor_stats_t stats;
+ size_t j;
+
+ stats = dfor_calc_stats(*dfor);
+ dfor->nbytes = dfor_calc_nbytes(*dfor);
+
+ if (bufSize != 0)
+ {
+ /* Max size of the dfor->pack is cnt * sizeof(size_t) */
+ dfor->pack = buf;
+ dfor->outer_mem = true;
+ if (dfor->nbytes > cnt * sizeof(size_t))
+ goto dfor_pack_error;
+ }
+ else
+ {
+ /* If a buffer was not provided by caller we allocate it by
+ * ourselves
+ */
+ dfor->pack = (uint8_t *)DFOR_MALLOC((dfor->nbytes));
+
+ dfor->outer_mem = false;
+ }
+
+ if (dfor->pack == NULL)
+ goto dfor_pack_error;
+
+ memset(dfor->pack, 0, dfor->nbytes);
+
+ /* index of the next free bit to be used: */
+ d = 0; /* - by a delta */
+ e = stats.delta_pack_nbits; /* - by an exception */
+ p = e + stats.exc_pack_nbits; /* - by an exception position index */
+ /* A mask looks like 0001111. It is also the
+ * max value of a short delta */
+ mask = width_to_mask(dfor->delta_wid);
+
+ j = 0;
+ for (size_t i = 0; i < vDeltas.cnt; i++)
+ {
+ d = bitpack_pack(dfor->pack, d, vDeltas.m[i] & mask,
+ dfor->delta_wid);
+
+ if (vDeltas.m[i] > mask)
+ {
+ Assert(isExcUsage == DFOR_EXC_USE);
+ Assert(usvExcPos.m[j] == i);
+ Assert(j < usvExcPos.cnt);
+ Assert(j < vExcPosDeltas.cnt);
+ Assert(dfor->exc_wid != 0);
+ Assert(dfor->exc_pos_wid != 0);
+
+ e = bitpack_pack(dfor->pack, e, vDeltas.m[i] >> dfor->delta_wid,
+ dfor->exc_wid);
+ p = bitpack_pack(dfor->pack, p, vExcPosDeltas.m[j], dfor->exc_pos_wid);
+ j++;
+ }
+ }
+
+ if (isExcUsage == DFOR_EXC_USE)
+ Assert(j == usvExcPos.cnt);
+ else
+ Assert(j == 0);
+
+ Assert(d == stats.delta_pack_nbits);
+ Assert(e == stats.delta_pack_nbits + stats.exc_pack_nbits);
+ Assert(p ==
+ stats.delta_pack_nbits + stats.exc_pack_nbits +
+ stats.exc_pos_pack_nbits);
+ res = 0;
+ }
+dfor_pack_ret:
+ vect_clear(&usvExcPos);
+ vect_clear(&vExcPosDeltas);
+ vect_clear(&vDeltas);
+ return res;
+dfor_pack_error:
+ if (dfor != NULL)
+ dfor_clear_meta(dfor);
+ res = -1;
+ goto dfor_pack_ret;
+}
+
+/*
+ * dfor_unpack
+ *
+ * If a caller needs to avoid using dynamic memory allocation, they have to:
+ * 1) provide the external memory buffer. The size of this buffer should be not
+ * less than:
+ * 2 * dfor.item_cnt * sizeof(item_t) + 2 * dfor.exc_cnt * sizeof(item_t)
+ *
+ * 2) the vVals vector has to be created but must not be initialised. The
+ * dfor_unpack sets vVals in the 'outer memory' regimen and will set vVal->m
+ * to buf.
+ *
+ * Provided dynamic allocation is used by the dfor_unpack, a caller will have to
+ * free the piece of memory pointed by vVals->m, using vect_clear(&vVals).
+ *
+ * Are the outer memory is used
+ */
+int
+dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals, size_t bufSize,
+ uint8_t buf[])
+{
+ int res = -1;
+ size_t szDeltaPack;
+ vect_t vExcs = { 0 };
+ vect_t vExcPoss = { 0 };
+ excalg_t isExcUsage = (dfor->exc_cnt == 0) ? DFOR_EXC_DONT_USE :
+ DFOR_EXC_USE;
+
+ if (vVals == NULL)
+ goto dfor_unpack_error;
+
+ if (bufSize != 0 &&
+ bufSize < (2 * dfor->item_cnt * sizeof(item_t) +
+ 2 * dfor->exc_cnt * sizeof(item_t)))
+ goto dfor_unpack_error;
+
+ szDeltaPack = dfor->delta_wid * dfor->item_cnt;
+
+ {
+ uint8_t *valsBuf = NULL;
+ if (bufSize != 0)
+ valsBuf = buf;
+
+ if (vect_init(vVals, dfor->item_cnt, (item_t *)valsBuf) != 0)
+ goto dfor_unpack_error;
+ }
+
+ /* Calculate exceptions */
+ if (isExcUsage == DFOR_EXC_USE)
+ {
+ size_t szExcPack;
+ size_t crExc; /* caret (cursor) */
+ size_t crPos; /* caret (cursor) */
+
+ uint8_t *excBuf = NULL;
+ uint8_t *excPossBuf = NULL;
+
+ int res1 = 0;
+ int res2 = 0;
+
+ szExcPack = dfor->exc_cnt * dfor->exc_wid;
+ crExc = szDeltaPack;
+ crPos = crExc + szExcPack;
+
+ if (bufSize != 0)
+ {
+ /* step over the memory occupied by vVals */
+ excBuf = buf + dfor->item_cnt * sizeof(item_t);
+ excPossBuf = excBuf + dfor->exc_cnt * sizeof(item_t);
+ }
+
+ res1 = vect_init(&vExcs, dfor->exc_cnt, (item_t *)excBuf);
+ res2 = vect_init(&vExcPoss, dfor->exc_cnt, (item_t *)excPossBuf);
+
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+
+
+ for (size_t posExc = 0, j = 0; j < dfor->exc_cnt; j++)
+ {
+ item_t deltaPos;
+ res1 = vect_append(&vExcs,
+ bitpack_unpack(dfor->pack, &crExc,
+ dfor->exc_wid));
+ /* Calculate the position of the exception from the delta of the
+ * position of the exception */
+ deltaPos = bitpack_unpack(dfor->pack, &crPos, dfor->exc_pos_wid);
+ posExc += deltaPos;
+ res2 = vect_append(&vExcPoss, posExc);
+ if (res1 != 0 || res2 != 0)
+ goto dfor_unpack_error;
+ }
+ Assert(crExc == szDeltaPack + szExcPack);
+ Assert(crPos ==
+ szDeltaPack + szExcPack + dfor->exc_pos_wid * dfor->exc_cnt);
+ }
+
+ { /* Unpack deltas and calculate target values */
+ item_t delta;
+ bool negDelta = false;
+ item_t prev = 0;
+ size_t j = 0; /* index of an exception and its position in vectors */
+ size_t crDelta = 0;
+
+ size_t maxItemWidth;
+ item_t maxItemMask;
+
+ maxItemWidth = sizeof(item_t) * 8;
+ maxItemMask = width_to_mask(maxItemWidth);
+
+ for (size_t i = 0; i < dfor->item_cnt; i++)
+ {
+ delta = bitpack_unpack(dfor->pack, &crDelta, dfor->delta_wid);
+
+ if (isExcUsage == DFOR_EXC_USE &&
+ j < vExcs.cnt &&
+ i == vExcPoss.m[j])
+ {
+ Assert(j < dfor->exc_cnt);
+ delta |= vExcs.m[j] << dfor->delta_wid;
+ j++;
+ }
+
+ if(dfor->signed_deltas)
+ {
+ negDelta = delta & 0x01;
+ delta >>= 1;
+ if ((negDelta && prev < delta) ||
+ (!negDelta && delta > maxItemMask - prev))
+ {
+ delta = maxItemMask - delta + 1;
+ negDelta = !negDelta;
+ }
+ }
+
+ if (negDelta)
+ prev -= delta;
+ else
+ prev += delta;
+
+ vect_append(vVals, prev);
+ }
+ Assert(crDelta == szDeltaPack);
+ res = 0;
+ }
+
+dfor_unpack_ret:
+ vect_clear(&vExcPoss);
+ vect_clear(&vExcs);
+ return res;
+dfor_unpack_error:
+ vect_clear(vVals);
+ res = -1;
+ goto dfor_unpack_ret;
+}
+
+void
+dfor_clear_meta(dfor_meta_t *meta)
+{
+ if (meta == NULL)
+ return;
+
+ if (meta->pack != NULL && !meta->outer_mem)
+ DFOR_FREE(meta->pack);
+
+ memset(meta, 0, sizeof(dfor_meta_t));
+}
+
+dfor_stats_t
+dfor_calc_stats(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ size_t nbytes;
+ stat.delta_pack_nbits = dfor.delta_wid * dfor.item_cnt;
+ stat.exc_pack_nbits = dfor.exc_wid * dfor.exc_cnt;
+ stat.exc_pos_pack_nbits = dfor.exc_pos_wid * dfor.exc_cnt;
+
+ stat.nbits = stat.delta_pack_nbits + stat.exc_pack_nbits + stat.exc_pos_pack_nbits;
+
+ /* If the division results in the remainder, we use an additional
+ * byte */
+ nbytes = (stat.nbits + 7) / 8;
+ stat.ratio = (float)(sizeof(item_t) * dfor.item_cnt) / (float)nbytes;
+
+ return stat;
+}
+
+size_t dfor_calc_nbytes(dfor_meta_t dfor)
+{
+ dfor_stats_t stat;
+ stat = dfor_calc_stats(dfor);
+ return (stat.nbits + 7) / 8;
+}
+
+#include "lib/dfor_templ_undef.h"
diff --git a/src/backend/lib/dfor_u16.c b/src/backend/lib/dfor_u16.c
new file mode 100644
index 00000000000..f7051f55925
--- /dev/null
+++ b/src/backend/lib/dfor_u16.c
@@ -0,0 +1,8 @@
+/*
+ * File: dfor_u16.c
+ */
+
+/* clang-format off */
+#include "lib/dfor_u16_config.h"
+#include "dfor_templ.c"
+/* clang-format on */
diff --git a/src/backend/lib/meson.build b/src/backend/lib/meson.build
index 0984bd0e3f6..7f6730efba1 100644
--- a/src/backend/lib/meson.build
+++ b/src/backend/lib/meson.build
@@ -2,6 +2,7 @@
dfor_sources = files(
'bitpack_u16.c',
+ 'dfor_u16.c',
'vect_u16.c'
)
diff --git a/src/include/lib/dfor_templ.h b/src/include/lib/dfor_templ.h
new file mode 100644
index 00000000000..8aa497dfaae
--- /dev/null
+++ b/src/include/lib/dfor_templ.h
@@ -0,0 +1,27 @@
+/*
+ * File: dfor_templ.h
+ */
+#include "dfor_templ_staple.h"
+
+extern int dfor_calc_deltas(size_t cnt, const item_t arr[], vect_t *vDeltas,
+ uniqsortvect_t *usvDeltaWidths,
+ vect_t *vWidthCounters, bool *sign);
+
+extern int dfor_calc_width(size_t cntDelta,
+ const uniqsortvect_t *usvDeltaWidths,
+ const vect_t *vWidthCounters, size_t *width,
+ size_t *cntExceptions);
+
+extern int dfor_pack(size_t cnt, const item_t arr[], excalg_t isExcUsage,
+ dfor_meta_t *dfor, size_t bufSize, uint8_t buf[]);
+
+extern int dfor_unpack(const dfor_meta_t *dfor, uniqsortvect_t *vVals,
+ size_t bufSize, uint8_t buf[]);
+
+extern void dfor_clear_meta(dfor_meta_t *dfor);
+
+extern dfor_stats_t dfor_calc_stats(dfor_meta_t dfor);
+
+extern size_t dfor_calc_nbytes(dfor_meta_t dfor);
+
+#include "dfor_templ_undef.h"
diff --git a/src/include/lib/dfor_templ_staple.h b/src/include/lib/dfor_templ_staple.h
new file mode 100644
index 00000000000..e47f6cb3a33
--- /dev/null
+++ b/src/include/lib/dfor_templ_staple.h
@@ -0,0 +1,126 @@
+/*
+ * File: dfor_staple_templ.h
+ */
+
+#include "c.h"
+
+/******************************************************************************
+ * NONREUSABLE CODE, allowed to be included only once
+ *
+ * This code should appear only once in a file (or in a chain of files)
+ * including this header with #include. So, we protect it from duplicates by
+ * means of macro
+ *
+ *****************************************************************************/
+#ifndef _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+#define _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_
+
+typedef struct {
+ size_t item_cnt;
+ size_t delta_wid;
+ size_t exc_cnt;
+ size_t exc_wid;
+ size_t exc_pos_wid;
+ size_t nbytes; /* size of pack in bytes */
+ uint8_t *pack;
+ bool outer_mem;
+ bool signed_deltas;
+} dfor_meta_t;
+
+typedef struct {
+ size_t nbits; /* size of pack in bits used in fact */
+ size_t delta_pack_nbits; /* in bits */
+ size_t exc_pack_nbits; /* in bits */
+ size_t exc_pos_pack_nbits; /* in bits */
+ float ratio; /* compression ratio */
+} dfor_stats_t;
+
+typedef enum {
+ DFOR_EXC_DONT_USE = 0,
+ DFOR_EXC_USE = 1
+} excalg_t;
+
+#endif /* _DFOR_STAPLE_TEMPL_H_UNIQUE_CODE_ */
+
+/******************************************************************************
+ * End of Code, allowed to be included only once
+ *****************************************************************************/
+
+/******************************************************************************
+ * REUSABLE CODE
+ *
+ * This code can be reused with #include directive in a file if DFOR_MARKER is
+ * redefined. This allows creation of several types of vectors with different
+ * types of members.
+ *
+ *****************************************************************************/
+
+#ifndef DFOR_ITEM_TYPE
+#error "DFOR_ITEM_TYPE macro is indefined."
+#endif
+
+#ifndef DFOR_MARKER
+#error "DFOR_MARKER macro is indefined."
+#endif
+
+#ifndef DFOR_MALLOC
+#error "DFOR_MALLOC macro is indefined."
+#endif
+
+#ifndef DFOR_FREE
+#error "DFOR_FREE macro is indefined."
+#endif
+
+#define MAKE_HEADER_NAME(v, m) CppAsString2(CppConcat2(v, m).h)
+
+/*
+ * Headers from vect and bitpack units
+ *
+ * Example: dfor_u16.c and dfor_u16.h need vect_u16.h and bitpack_u16.h
+ */
+#include MAKE_HEADER_NAME(lib/vect_, DFOR_MARKER)
+#include MAKE_HEADER_NAME(lib/bitpack_, DFOR_MARKER)
+
+/* Types */
+#define item_t DFOR_ITEM_TYPE
+#define vect_t CppConcatTriple2(vect_, DFOR_MARKER, _t)
+#define uniqsortvect_t CppConcatTriple2(uniqsortvect_, DFOR_MARKER, _t)
+
+/* Functions */
+#define dfor_calc_deltas CppConcatTriple2(dfor_, DFOR_MARKER, _calc_deltas)
+#define dfor_calc_width CppConcatTriple2(dfor_, DFOR_MARKER, _calc_width)
+#define dfor_pack CppConcatTriple2(dfor_, DFOR_MARKER, _pack)
+#define dfor_unpack CppConcatTriple2(dfor_, DFOR_MARKER, _unpack)
+#define dfor_analyze CppConcatTriple2(dfor_, DFOR_MARKER, _analyze)
+#define dfor_clear_meta CppConcatTriple2(dfor_, DFOR_MARKER, _clear_meta)
+#define dfor_calc_stats CppConcatTriple2(dfor_, DFOR_MARKER, _calc_stats)
+#define dfor_calc_nbytes CppConcatTriple2(dfor_, DFOR_MARKER, _calc_nbytes)
+
+/* Functions of the vect unit */
+#define vect_init CppConcatTriple2(vect_, DFOR_MARKER, _init)
+#define vect_fill CppConcatTriple2(vect_, DFOR_MARKER, _fill)
+#define vect_reserve CppConcatTriple2(vect_, DFOR_MARKER, _reserve)
+#define vect_append CppConcatTriple2(vect_, DFOR_MARKER, _append)
+#define vect_print CppConcatTriple2(vect_, DFOR_MARKER, _print)
+#define vect_compare CppConcatTriple2(vect_, DFOR_MARKER, _compare)
+#define vect_insert CppConcatTriple2(vect_, DFOR_MARKER, _insert)
+#define vect_clear CppConcatTriple2(vect_, DFOR_MARKER, _clear)
+
+#define usv_insert CppConcatTriple2(usv_, DFOR_MARKER, _insert)
+#define usv_search CppConcatTriple2(usv_, DFOR_MARKER, _search)
+
+/* Functions of the bitpack unit */
+#define width_from_val CppConcatTriple2(width_, DFOR_MARKER, _from_val)
+#define width_to_mask CppConcatTriple2(width_, DFOR_MARKER, _to_mask)
+#define bitpack_pack CppConcatTriple2(bitpack_, DFOR_MARKER, _pack)
+#define bitpack_unpack CppConcatTriple2(bitpack_, DFOR_MARKER, _unpack)
+
+/******************************************************************************
+ * End of Reusable Code
+ *****************************************************************************/
+
+/*
+ * Don't forget to include this code in your file that uses this header
+ * #include "dfor_templ_undef.h"
+ *
+ */
diff --git a/src/include/lib/dfor_templ_undef.h b/src/include/lib/dfor_templ_undef.h
new file mode 100644
index 00000000000..7a22ea9771f
--- /dev/null
+++ b/src/include/lib/dfor_templ_undef.h
@@ -0,0 +1,28 @@
+#undef item_t
+#undef vect_t
+#undef uniqsortvect_t
+
+#undef dfor_calc_deltas
+#undef dfor_calc_width
+#undef dfor_pack
+#undef dfor_unpack
+#undef dfor_analyze
+#undef dfor_calc_stats
+#undef dfor_calc_nbytes
+
+#undef vect_init
+#undef vect_fill
+#undef vect_reserve
+#undef vect_append
+#undef vect_print
+#undef vect_compare
+#undef vect_insert
+#undef vect_clear
+
+#undef usv_insert
+#undef usv_search
+
+#undef width_from_val
+#undef width_to_mask
+#undef bitpack_pack
+#undef bitpack_unpack
diff --git a/src/include/lib/dfor_u16.h b/src/include/lib/dfor_u16.h
new file mode 100644
index 00000000000..716c99dbc55
--- /dev/null
+++ b/src/include/lib/dfor_u16.h
@@ -0,0 +1,13 @@
+/*
+ * File: dfor_u16.h
+ */
+
+#ifndef _DFOR_U16_H_
+#define _DFOR_U16_H_
+
+/* clang-format off */
+#include "dfor_u16_config.h"
+#include "dfor_templ.h"
+/* clang-format on */
+
+#endif /* _DFOR_U16_H_ */
diff --git a/src/include/lib/dfor_u16_config.h b/src/include/lib/dfor_u16_config.h
new file mode 100644
index 00000000000..751937ac513
--- /dev/null
+++ b/src/include/lib/dfor_u16_config.h
@@ -0,0 +1,4 @@
+#define DFOR_ITEM_TYPE uint16_t
+#define DFOR_MARKER u16
+#define DFOR_MALLOC malloc
+#define DFOR_FREE free
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index e979d0a7436..0ab8ad276c7 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -163,6 +163,10 @@ do
test "$f" = src/include/lib/bitpack_templ_staple.h && continue
test "$f" = src/include/lib/bitpack_templ_undef.h && continue
+ test "$f" = src/include/lib/dfor_templ.h && continue
+ test "$f" = src/include/lib/dfor_templ_staple.h && continue
+ test "$f" = src/include/lib/dfor_templ_undef.h && continue
+
test "$f" = src/include/lib/vect_templ.h && continue
test "$f" = src/include/lib/vect_templ_staple.h && continue
test "$f" = src/include/lib/vect_templ_undef.h && continue
--
2.53.0
[text/x-patch] v13-0004-Tests-for-Delta-Frame-of-Reference-unit.patch (23.4K, 5-v13-0004-Tests-for-Delta-Frame-of-Reference-unit.patch)
download | inline diff:
From 582a7e5997e9d7d2fe7c31829d8515921fba58ad Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Mon, 13 Apr 2026 23:18:07 +0800
Subject: [PATCH v13 4/5] Tests for Delta Frame of Reference unit.
The unit test is implemented as a C program (ELF executable). The test
can be run with the 'make check-unit'. Tests support the TAP protocol
and are executed using the Prove utility.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/test/dfor/.gitignore | 1 +
src/test/dfor/Makefile | 3 +-
src/test/dfor/meson.build | 2 +
src/test/dfor/test_dfor_u16.c | 536 ++++++++++++++++++++++++++++++++++
4 files changed, 541 insertions(+), 1 deletion(-)
create mode 100644 src/test/dfor/test_dfor_u16.c
diff --git a/src/test/dfor/.gitignore b/src/test/dfor/.gitignore
index 0d77a51216b..447e95c0c09 100644
--- a/src/test/dfor/.gitignore
+++ b/src/test/dfor/.gitignore
@@ -1,3 +1,4 @@
test_bitpack_u16
+test_dfor_u16
test_uniqsortvect_u16
test_vect_u16
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 4fc9f4bc1ba..2ca98f76a0f 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -33,7 +33,8 @@ LIBTAP_OBJS = $(top_builddir)/src/test/libtap/tap.o
TESTS= test_vect_u16 \
test_uniqsortvect_u16 \
- test_bitpack_u16
+ test_bitpack_u16 \
+ test_dfor_u16
$(TESTS:%=%.o): CPPFLAGS += -I$(top_builddir)/src/test -DFRONTEND
diff --git a/src/test/dfor/meson.build b/src/test/dfor/meson.build
index ce762c52430..4a760ab68fa 100644
--- a/src/test/dfor/meson.build
+++ b/src/test/dfor/meson.build
@@ -8,6 +8,7 @@ dfor_dir = join_paths(meson.project_source_root(), 'src/backend/lib')
dfor_sources = files(
join_paths(dfor_dir, 'vect_u16.c'),
join_paths(dfor_dir, 'bitpack_u16.c'),
+ join_paths(dfor_dir, 'dfor_u16.c'),
)
dfor_test_lib = static_library(
@@ -36,6 +37,7 @@ test_names = [
'test_vect_u16',
'test_uniqsortvect_u16',
'test_bitpack_u16',
+ 'test_dfor_u16',
]
foreach t : test_names
diff --git a/src/test/dfor/test_dfor_u16.c b/src/test/dfor/test_dfor_u16.c
new file mode 100644
index 00000000000..efa91e3e6c8
--- /dev/null
+++ b/src/test/dfor/test_dfor_u16.c
@@ -0,0 +1,536 @@
+/*
+ * test_dfor.c
+ */
+
+#include "lib/bitpack_u16.h"
+#include "lib/dfor_u16.h"
+#include "lib/vect_u16.h"
+#include "libtap/tap.h"
+#include "test.h"
+
+void test_delta_calculation(size_t cnt, uint16_t inArr[], size_t cntDelta,
+ uint16_t marDeltasExpected[], size_t cntWidth,
+ uint16_t marWidthsExpected[], size_t cntStat,
+ uint16_t marWidthsStatExpected[],
+ bool deltaSinednessRequested,
+ bool deltaSinednessExpected);
+
+void test_calc_exceptions(size_t numDeltas, size_t numWidths,
+ uint16_t marWidths[], size_t numCounts,
+ uint16_t marCounts[], size_t szAwaitedWidth,
+ size_t cntAwaitedExcCount);
+
+void test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[]);
+
+void
+test_delta_calculation(size_t cnt, uint16_t inArr[],
+ size_t cntDelta, uint16_t marDeltasExpected[],
+ size_t cntWidth, uint16_t marWidthsExpected[],
+ size_t cntStat, uint16_t marWidthsStatExpected[],
+ bool deltaSignednessRequested,
+ bool deltaSignednessExpected)
+{
+ vect_u16_t vDeltas;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ vect_u16_t awaited;
+ int res;
+ bool deltaSignedness;
+
+ printf("------------------------------------------------\n");
+ printf("Test\n");
+ printf("------------------------------------------------\n");
+
+ printf(" inArr:");
+ for (size_t i = 0; i < cnt; i++)
+ printf(" %u", (uint32_t)inArr[i]);
+ printf("\n");
+
+ vect_u16_init(&vDeltas, 0, NULL);
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_init(&vWidthCounters, 0, NULL);
+
+ deltaSignedness = deltaSignednessRequested;
+ /* Tested function */
+ res = dfor_u16_calc_deltas(cnt, inArr, &vDeltas, &usvDeltaWidths,
+ &vWidthCounters, &deltaSignedness);
+ cmp_ok(res, "==", 0);
+
+ printf(" Delta expected:");
+ for (size_t i = 0; i < cntDelta; i++)
+ printf(" %u", (uint32_t)marDeltasExpected[i]);
+ printf("\n");
+
+ printf(" Delta fact: ");
+ vect_u16_print(&vDeltas);
+
+ cmp_ok(vDeltas.cnt, "==", cnt, "The Delta count is OK.");
+ vect_u16_init(&awaited, 0, NULL);
+ vect_u16_fill(&awaited, cnt, marDeltasExpected);
+ cmp_ok(deltaSignedness, "==", deltaSignednessExpected, "The signedness result is OK.");
+ cmp_ok(vect_u16_compare(&vDeltas, &awaited), "==", 0,
+ "All deltas are calculated properly");
+ vect_u16_clear(&awaited);
+
+ printf(" Width expected:");
+ for (size_t i = 0; i < cntWidth; i++)
+ printf(" %u", (uint32_t)marWidthsExpected[i]);
+ printf("\n");
+
+ printf(" Width fact: ");
+ vect_u16_print(&usvDeltaWidths);
+
+ cmp_ok(usvDeltaWidths.cnt, "==", cntWidth, "The Width count is OK.");
+
+ /* don't really need initialisation after vect_clean having been done
+ * above*/
+ /* vect_u16_init(&awaited, 0, NULL); */
+
+ vect_u16_fill(&awaited, cntWidth, marWidthsExpected);
+ cmp_ok(vect_u16_compare(&usvDeltaWidths, &awaited), "==", 0,
+ "All delta widths is OK.");
+ vect_u16_clear(&awaited);
+
+ printf(" Deltas statistics expected:");
+ for (size_t i = 0; i < cntStat; i++)
+ printf(" %u", (uint32_t)marWidthsStatExpected[i]);
+ printf("\n");
+
+ printf(" Deltas statistics statistics fact: ");
+ vect_u16_print(&vWidthCounters);
+
+ cmp_ok(
+ usvDeltaWidths.cnt, "==", vWidthCounters.cnt,
+ "The count of statistics of widths is equal to the count of widths.");
+
+ /* don't really need initialisation after vect_clean has been done
+ * above*/
+ vect_u16_fill(&awaited, cntStat, marWidthsStatExpected);
+ cmp_ok(vect_u16_compare(&vWidthCounters, &awaited), "==", 0,
+ "Width statistics is OK.");
+ vect_u16_clear(&awaited);
+
+ vect_u16_clear(&vDeltas);
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_calc_exceptions(size_t numDeltas, size_t numWidths, uint16_t marWidths[],
+ size_t numCounts, uint16_t marCounts[],
+ size_t szAwaitedWidth, size_t cntAwaitedExcCount)
+{
+ int res;
+ vect_u16_t vWidthCounters;
+ uniqsortvect_u16_t usvDeltaWidths;
+ size_t width, cntExceptions;
+
+ vect_u16_init(&usvDeltaWidths, 0, NULL);
+ vect_u16_fill(&usvDeltaWidths, numWidths, marWidths);
+
+ vect_u16_init(&vWidthCounters, 0, NULL);
+ vect_u16_fill(&vWidthCounters, numCounts, marCounts);
+
+ res = dfor_u16_calc_width(numDeltas, &usvDeltaWidths, &vWidthCounters,
+ &width, &cntExceptions);
+ cmp_ok(res, "==", 0);
+ cmp_ok(width, "==", szAwaitedWidth, "Width is OK.");
+ cmp_ok(cntExceptions, "==", cntAwaitedExcCount, "Exceptions num is OK");
+
+ vect_u16_clear(&usvDeltaWidths);
+ vect_u16_clear(&vWidthCounters);
+}
+
+void
+test_dfor(size_t cnt, uint16_t arr[], excalg_t isExcUsage,
+ size_t widDeltaWidthAwaited, size_t cntExcCntAwaited,
+ size_t widExcWidAwaited, size_t widExcPosWidAwaited,
+ size_t cntBitsCountAwaited, size_t cntByteCountAwaited,
+ float flMinRatioAwaited, uint8_t u8arPackAwaited[])
+{
+ int res;
+ dfor_meta_t dfor;
+ dfor_stats_t stats;
+ uniqsortvect_u16_t extracted;
+
+ res = dfor_u16_pack(cnt, arr, isExcUsage, &dfor, 0, NULL);
+ cmp_ok(res, "==", 0, "dfor_pack func has processed OK.");
+ cmp_ok(dfor.item_cnt, "==", cnt, "Count of deltas is OK.");
+ cmp_ok(dfor.delta_wid, "==", widDeltaWidthAwaited, "Delta width is OK.");
+ cmp_ok(dfor.exc_cnt, "==", cntExcCntAwaited, "Exception count is OK.");
+ cmp_ok(dfor.exc_wid, "==", widExcWidAwaited, "Exception width is OK.");
+ cmp_ok(dfor.exc_pos_wid, "==", widExcPosWidAwaited,
+ "Exception position width is OK.");
+ ok(dfor.pack != NULL, "Pack is created (not NULL).");
+
+ stats = dfor_u16_calc_stats(dfor);
+ cmp_ok(stats.nbits, "==", cntBitsCountAwaited, "Bits count is OK.");
+ cmp_ok(dfor.nbytes, "==", cntByteCountAwaited, "Bytes count is OK.");
+ ok(stats.ratio > flMinRatioAwaited, "Compression ratio is OK.");
+
+ if (u8arPackAwaited != NULL)
+ ok(0 == memcmp(dfor.pack, u8arPackAwaited, cntByteCountAwaited),
+ "Pack content is OK.");
+ else
+ ok(0 == 0, "Pack content check is skipped.");
+
+ test_print_u16_array(cnt, (uint16_t *)arr, "\n\nOriginal integer array");
+ test_print_u8_array(dfor.nbytes, dfor.pack, "Compressed integer array");
+ printf("Compression ratio:%f\n\n", stats.ratio);
+
+ vect_u16_init(&extracted, 0, NULL);
+
+ dfor_u16_unpack(&dfor, &extracted, 0, NULL);
+ cmp_ok(extracted.cnt, "==", cnt, "Extracted count is OK");
+ cmp_ok(0, "==", memcmp(arr, extracted.m, cnt),
+ "Extracted array is equal to original");
+
+ free(dfor.pack);
+ vect_u16_clear(&extracted);
+}
+
+int
+main(void)
+{
+ plan(267);
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for sorted sequences \n");
+ {
+ /* Sorted sequences */
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1 }, /* marDeltasExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19 }, /* inArr */
+ 10,
+ (uint16_t[]) { 1, 2, 2, 2, 2, 2, 2, 2, 2, 2 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 2 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 416 }, /* inArr */
+ 14,
+ (uint16_t[]) { 100, 100, 100, 100, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2 },
+ /* marDeltasExpected*/
+ 3, (uint16_t[]) { 1, 2, 7 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 123 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 7 }, /* marWidthsExpected */
+ 1,
+ (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for sorted sequences PASSED \n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for unsorted sequences and test SIGNEDNESS \n");
+
+ /* Unsorted sequences */
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0x0, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 10 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, +1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 7 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 2, 2, 2, 2, 2, 2, 2, 2, 3 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 11 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, -1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 7 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0, 2, 2, 2, 2, 2, 2, 2, 2, 3 }, /* marDeltasExpected*/
+ /*BIN: 00, 10, 10, 10, 10, 10, 10, 10, 10, 11 */
+ /*DEC: +0, +1, +1, +1, +1, +1, +1, +1, +1, -1 */
+ 1, (uint16_t[]) { 2 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 10 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(
+ 10, (uint16_t[]) { 1, 3, 5, 3, 5, 7, 9, 11, 13, 15 }, /* inArr */
+ 10,
+ (uint16_t[]) { 0x2, 0x4, 0x4, 0x5, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4 }, /* marDeltasExpected*/
+ /*BIN: 10, 100, 100, 101, 100, 100, 100, 100, 100, 100 */
+ /*DEC: +1, +2, +2, -2, +2, +2, +2, +2, +2, +2 */
+ 2, (uint16_t[]) { 2, 3 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 9 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+ test_delta_calculation(
+ 14,
+ (uint16_t[]) { 100, 200, 300, 400, 401, 402, 403, 404, 406, 408,
+ 410, 412, 414, 412 }, /* inArr */
+ 14,
+ (uint16_t[]) { 200, 200, 200, 200, 2, 2, 2, 2, 4, 4, 4, 4, 4, 5 }, /* marDeltasExpected code*/
+ /* DEC: +100, +100, +100, +1, +1, +1, +1, +1, +2, +2, +2, +2, +2, -2 */
+ 3, (uint16_t[]) { 2, 3, 8 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 4, 6, 4 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 123 }, /* inArr */
+ 1, (uint16_t[]) { 246 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 8 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(0, NULL, /* inArr */
+ 0, NULL, /* marDeltasExpected*/
+ 0, NULL, /* marWidthsExpected */
+ 0, NULL, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for unsorted sequences and test SIGNEDNESS PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA CALCULATION for unsorted sequences and test OVERLAPPING trick \n");
+
+ test_delta_calculation(1, (uint16_t[]) { 33000 }, /* inArr */
+ 1, (uint16_t[]) { 33000 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(1, (uint16_t[]) { 33000 }, /* inArr */
+ /* 65535 - 33000 + 1 = 32536 = 0x7F18
+ -32536 = (0x7F18 << 1) + 1 = 0xFE31 = 65073 */
+ 1, (uint16_t[]) { 0xFE31 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 1 }, /* marWidthsStatExpected */
+ true, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(2, (uint16_t[]) { 0, 33000 }, /* inArr */
+ 2, (uint16_t[]) { 0, 33000 }, /* marDeltasExpected*/
+ 2, (uint16_t[]) { 1, 16 }, /* marWidthsExpected */
+ 2, (uint16_t[]) { 1, 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ false /* deltaSignedness expected */);
+
+ test_delta_calculation(2, (uint16_t[]) { 33000, 0 }, /* inArr */
+ /* 65535 - 33000 + 1 = 32536 = 0x7F18;
+ (0x7F18 << 1) = 0xFE30 = 65072;
+ -32536 = (0x7F18 << 1) | 1 = 0xFE31 = 65073 */
+ 2, (uint16_t[]) { 0xFE31, 0xFE30 }, /* marDeltasExpected*/
+ 1, (uint16_t[]) { 16 }, /* marWidthsExpected */
+ 1, (uint16_t[]) { 2 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ test_delta_calculation(16, (uint16_t[]) { 0, 1, 0, 3, 4, 5, 6, 7,
+ 8, 9, 10, 11, 12, 13, 14, 65534 }, /* inArr */
+ 16, (uint16_t[]) { 0x0, 0x2, 0x3, 0x6, 0x2, 0x2, 0x2, 0x2,
+ 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x2, 0x21 }, /* marDeltasExpected*/
+ 3, (uint16_t[]) { 2, 3, 6 }, /* marWidthsExpected */
+ 3, (uint16_t[]) { 14, 1, 1 }, /* marWidthsStatExpected */
+ false, /* deltaSignedness requested */
+ true /* deltaSignedness expected */);
+
+ printf("Test DELTA CALCULATION for unsorted sequences and test OVERLAPPING trick PASSED \n");
+ printf("========================================\n\n");
+ }
+
+ printf("========================================\n");
+ printf("Test EXCEPTIONS CALCULATION\n");
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths of deltas*/
+ 3, (uint16_t[]) { 4, 6, 4 }, /* statistics on widths */
+ 7, /* width of short deltas */
+ 0); /* number of exceptions*/
+
+ test_calc_exceptions(14, 3, (uint16_t[]) { 1, 2, 7 }, /* widths */
+ 3, (uint16_t[]) { 6, 7, 1 }, /* stat */
+ 2, 1); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 3, (uint16_t[]) { 5, 6, 12 }, /* widths */
+ 3, (uint16_t[]) { 36, 2, 2 }, /* stat */
+ 5, 4); /* short deltas width, exceptions count*/
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 36, 1, 1, 2 }, 5, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 35, 1, 2, 2 }, 6, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 34, 1, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 34, 2, 3 }, 7, 3);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 33, 2, 4 }, 7, 4);
+
+ test_calc_exceptions(40, 4, (uint16_t[]) { 5, 6, 7, 12 }, 4,
+ (uint16_t[]) { 1, 32, 2, 5 }, 12, 0);
+
+ printf("Test EXCEPTIONS CALCULATION PASSED\n");
+ printf("========================================\n\n");
+
+ printf("========================================\n");
+ printf("Test DELTA FRAME OF REFERENCES PACKING\n");
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 16, /* awaited bits count */
+ 2, /* cntByteCountAwaited */
+ 15.99, /* awaited ratio min value (ratio >= 15.99) */
+ (uint8_t[]) { 0xFE, 0xFF });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 10, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 10 * 16, /* awaited bits count */
+ 20, /* awaited bytes count */
+ 1.5, /* awaited ratio min value */
+ (uint8_t[]) { 0x00, 0x04, 0x10, 0x40, 0x00, 0x01, 0x04,
+ 0x10, 0x40, 0x00, 0x01, 0x04, 0x10, 0x40,
+ 0x00, 0x01, 0x04, 0x10, 0x40, 0xFC });
+
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 1023 }, /* cnt, array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 1, /* awaited count of exceptions */
+ 9, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 16 * 1 + 9 + 4, /* awaited bits count */
+ 4, /* awaited bytes count */
+ 7.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFF, 0xF8, 0x1F });
+
+ test_dfor(30, /* cnt */
+ (uint16_t[]) { 0, 1, 2, 3, 4,
+ 5, 6, 7, 8, 9,
+ /* delta=2, pos=10, deltapos=10 */ 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25,
+ /* delta=3, pos=25, deltapos=15 */ 28, 29, 30, 31,
+ /* delta=3, pos=29, deltapos=4 */ 34 }, /* array */
+ DFOR_EXC_USE, /* flag on use of exceptions */
+ 1, /* awaited width of short deltas */
+ 3, /* awaited count of exceptions */
+ 1, /* awaited exception width*/
+ 4, /* awaited exception position width*/
+ 30 * 1 + 3 * 1 + 3 * 4, /* awaited bits count */
+ 6, /* awaited bytes count */
+ 9.99, /* awaited ratio min value */
+ (uint8_t[]) { 0xFE, 0xFB, 0xFF, 0xFF, 0xF5, 0x09 });
+
+ /* Unsorted sequences */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 13 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 2, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 32, /* awaited bits count */
+ 4, /* cntByteCountAwaited */
+ 7.99, /* awaited ratio min value (ratio >= 7.99) */
+ (uint8_t[]) { 0xA8, 0xAA, 0xAA, 0xEA });
+
+ /* Test the decoding algorithm for the overlap trick */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 65535, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+ 15 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 3, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 48, /* awaited bits count */
+ 6, /* cntByteCountAwaited */
+ 4, /* awaited ratio min value */
+ (uint8_t[]) { 0x50, 0x4d, 0x49, 0x92, 0x24, 0x49 });
+
+ /* The next test does not result in activation of the overlap trick since
+ the sequence is sorted. */
+ test_dfor(16,
+ (uint16_t[]) { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 65535 }, /* cnt, array */
+ DFOR_EXC_DONT_USE, /* flag on use of exceptions */
+ 16, /* awaited width of short deltas */
+ 0, /* awaited count of exceptions */
+ 0, /* awaited exception width*/
+ 0, /* awaited exception position width*/
+ 256, /* awaited bits count */
+ 32, /* cntByteCountAwaited */
+ 0.9999, /* awaited ratio min value = 1 */
+ (uint8_t[]) { 00, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 01, 00, 0xf1, 0xff });
+
+ printf("Test DELTA FRAME OF REFERENCES PACKING PASSED\n");
+ printf("========================================\n\n");
+
+ done_testing();
+}
--
2.53.0
[text/x-patch] v13-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch (41.1K, 6-v13-0005-Use-Delta-Frame-of-Reference-DFoR-to-compress-pr.patch)
download | inline diff:
From dc84023f5c8d6d43818f1cd37196fb03cd5e789d Mon Sep 17 00:00:00 2001
From: Evgeny Voropaev <[email protected]>
Date: Fri, 20 Mar 2026 17:30:22 +0800
Subject: [PATCH v13 5/5] Use Delta Frame of Reference (DFoR) to compress
prune/freeze records.
A prune/freeze record contains four sequences of integers representing
frozen, redirected, unused, and dead tuples. Using DFoR algorithms, the
`unused` and `dead` sequences are now compressed. The `frozen`
and `redirected` sequences cannot be compressed because the order of
their elements is significant, and DFoR does not support unsorted
sequences yet. The theoretical compression ratio for dfor_u16 can reach
up to 16.
The new GUC wal_prune_dfor_compression controls (enables or
disables) compression for prune/freeze records.
An integral TAP test, 052_prune_dfor_compression.pl, has been
implemented. It demonstrates an average compression ratio of at least 5
when analyzing prune/freeze records in practice.
Author: Evgeny Voropaev <[email protected]> <[email protected]>
Reviewed by: Andrey Borodin <[email protected]>
---
src/backend/access/heap/heapam_xlog.c | 21 +-
src/backend/access/heap/pruneheap.c | 209 +++++++++++--
src/backend/access/rmgrdesc/Makefile | 1 +
.../access/rmgrdesc/heapam_xlog_dfor.c | 114 +++++++
src/backend/access/rmgrdesc/heapdesc.c | 94 ++++--
src/backend/access/rmgrdesc/meson.build | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/guc_tables.c | 1 +
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/pg_waldump/.gitignore | 9 +
src/bin/pg_waldump/Makefile | 26 +-
src/bin/pg_waldump/meson.build | 1 +
src/include/access/heapam_xlog.h | 10 +-
src/include/access/heapam_xlog_dfor.h | 144 +++++++++
src/test/dfor/Makefile | 2 +
.../recovery/t/052_prune_dfor_compression.pl | 283 ++++++++++++++++++
16 files changed, 866 insertions(+), 59 deletions(-)
create mode 100644 src/backend/access/rmgrdesc/heapam_xlog_dfor.c
create mode 100644 src/include/access/heapam_xlog_dfor.h
create mode 100644 src/test/recovery/t/052_prune_dfor_compression.pl
diff --git a/src/backend/access/heap/heapam_xlog.c b/src/backend/access/heap/heapam_xlog.c
index 9ed7024e814..1d44cf96692 100644
--- a/src/backend/access/heap/heapam_xlog.c
+++ b/src/backend/access/heap/heapam_xlog.c
@@ -16,6 +16,7 @@
#include "access/bufmask.h"
#include "access/heapam.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/visibilitymap.h"
#include "access/xlog.h"
#include "access/xlogutils.h"
@@ -105,12 +106,26 @@ heap_xlog_prune_freeze(XLogReaderState *record)
OffsetNumber *frz_offsets;
char *dataptr = XLogRecGetBlockData(record, 0, &datalen);
bool do_prune;
+ char *cursor PG_USED_FOR_ASSERTS_ONLY;
- heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
+ /*
+ * Provide DFoR unpacking with outer buffers. 3 buffer parts are
+ * for saving the redirected, the dead and the unused tuple offsets.
+ * Additional three parts are for internal needs of the dfor_unpack
+ * function.
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[6 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
+ cursor = heap_xlog_deserialize_prune_and_freeze(dataptr, xlrec.flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf_aligned.dfor_buf);
do_prune = nredirected > 0 || ndead > 0 || nunused > 0;
@@ -156,7 +171,7 @@ heap_xlog_prune_freeze(XLogReaderState *record)
}
/* There should be no more data */
- Assert((char *) frz_offsets == dataptr + datalen);
+ Assert(cursor == dataptr + datalen);
/*
* The critical integrity requirement here is that we must never end
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 74c355be219..994e7b57c35 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -16,6 +16,7 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/transam.h"
@@ -238,7 +239,6 @@ static bool heap_page_will_freeze(bool did_tuple_hint_fpi, bool do_prune, bool d
static bool heap_page_will_set_vm(PruneState *prstate, PruneReason reason,
bool do_prune, bool do_freeze);
-
/*
* Optionally prune and repair fragmentation in the specified page.
*
@@ -2529,6 +2529,24 @@ heap_log_freeze_plan(HeapTupleFreeze *tuples, int ntuples,
return nplans;
}
+/*
+ * Comparator for offsets.
+ */
+static int
+offset_cmp(const void *arg1, const void *arg2)
+{
+ const OffsetNumber *offset1 = arg1;
+ const OffsetNumber *offset2 = arg2;
+ return (*offset1 > *offset2) - (*offset1 < *offset2);
+}
+
+#define ST_SORT sort_offsets
+#define ST_ELEMENT_TYPE_VOID
+#define ST_COMPARE(a, b) offset_cmp(a, b)
+#define ST_SCOPE static
+#define ST_DEFINE
+#include "lib/sort_template.h"
+
/*
* Write an XLOG_HEAP2_PRUNE* WAL record
*
@@ -2586,11 +2604,45 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
bool do_set_vm = vmflags & VISIBILITYMAP_VALID_BITS;
bool heap_fpi_allowed = true;
+ dfor_meta_t frozen_meta = {0};
+ dfor_meta_t redirected_meta = {0};
+ dfor_meta_t dead_meta = {0};
+ dfor_meta_t unused_meta = {0};
+
+ uint8 frozen_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 redirected_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 dead_meta_pack[MAX_PACKED_META_SIZE];
+ uint8 unused_meta_pack[MAX_PACKED_META_SIZE];
+
+ /*
+ * Since this code is run in a critical section we can't use dynamic
+ * allocation during DFoR packing, but we can use buffers allocated in the
+ * stack. We need at maximum:
+ * 1) 4 * DFOR_BUF_PART_SIZE
+ * - for 4 packed sequences: frozen, redirected, dead, unused
+ * 2) 3 * DFOR_BUF_PART_SIZE
+ * - for internal needs of the dfor_pack function.
+ *
+ * Overall, 7 * DFOR_BUF_PART_SIZE
+ */
+ union
+ {
+ int32 align_me; /* Forces 4-byte alignment */
+ uint8 dfor_buf[7 * DFOR_BUF_PART_SIZE];
+ } dfor_buf_aligned;
+
Assert((vmflags & VISIBILITYMAP_VALID_BITS) == vmflags);
xlrec.flags = 0;
regbuf_flags_heap = REGBUF_STANDARD;
+ /* Heuristically estimated threshold for turning on DFoR compression */
+ if (wal_prune_dfor_compression &&
+ (nredirected > 5 || ndead > 9 || nunused > 9 || nfrozen > 9))
+ {
+ xlrec.flags |= XLHP_DFOR_COMPRESSED;
+ }
+
/*
* We can avoid an FPI of the heap page if the only modification we are
* making to it is to set PD_ALL_VISIBLE and checksums/wal_log_hints are
@@ -2622,6 +2674,10 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
if (do_set_vm)
XLogRegisterBuffer(1, vmbuffer, 0);
+ /*
+ * xlhp_freeze_plans is array of structures and is not a sequence
+ * of integers, that is why we cannot use DFoR compression here.
+ */
if (nfrozen > 0)
{
int nplans;
@@ -2640,39 +2696,138 @@ log_heap_prune_and_freeze(Relation relation, Buffer buffer,
XLogRegisterBufData(0, plans,
sizeof(xlhp_freeze_plan) * nplans);
}
- if (nredirected > 0)
- {
- xlrec.flags |= XLHP_HAS_REDIRECTIONS;
- redirect_items.ntargets = nredirected;
- XLogRegisterBufData(0, &redirect_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, redirected,
- sizeof(OffsetNumber[2]) * nredirected);
- }
- if (ndead > 0)
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) != 0)
{
- xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+ int frozen_pack_res = 0;
+ int redirected_pack_res = 0;
+ int dead_pack_res = 0;
+ int unused_pack_res = 0;
+
+ if (nfrozen > 0)
+ {
+ frozen_pack_res = dfor_u16_pack(nfrozen, frz_offsets,
+ DFOR_EXC_USE, &frozen_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ 3 * DFOR_BUF_PART_SIZE);
+ }
+
+ if (nredirected > 0)
+ {
+ redirected_pack_res = dfor_u16_pack(nredirected * 2, redirected,
+ DFOR_EXC_USE, &redirected_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf);
+ }
+
+ if (ndead > 0)
+ {
+ sort_offsets(dead, ndead, sizeof(OffsetNumber));
+ dead_pack_res = dfor_u16_pack(ndead, dead, DFOR_EXC_USE, &dead_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ DFOR_BUF_PART_SIZE);
+ }
+
+ if (nunused > 0)
+ {
+ sort_offsets(unused, nunused, sizeof(OffsetNumber));
+ unused_pack_res = dfor_u16_pack(nunused, unused, DFOR_EXC_USE,
+ &unused_meta,
+ 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf_aligned.dfor_buf +
+ 2 * DFOR_BUF_PART_SIZE);
+ }
+
+ if (frozen_pack_res == 0 && redirected_pack_res == 0 &&
+ dead_pack_res == 0 && unused_pack_res == 0)
+ {
+ /* All stages of packing have succeeded. We can save DFoR packets
+ * into the log. */
+ size_t meta_pack_sz;
+
+ if (nfrozen > 0)
+ {
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &frozen_meta, frozen_meta_pack);
+
+ XLogRegisterBufData(0, &frozen_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, frozen_meta.pack, frozen_meta.nbytes);
+ }
+ if (nredirected > 0)
+ {
+ xlrec.flags |= XLHP_HAS_REDIRECTIONS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &redirected_meta, redirected_meta_pack);
+
+ XLogRegisterBufData(0, &redirected_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, redirected_meta.pack,
+ redirected_meta.nbytes);
+ }
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &dead_meta, dead_meta_pack);
+
+ XLogRegisterBufData(0, &dead_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, dead_meta.pack, dead_meta.nbytes);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
- dead_items.ntargets = ndead;
- XLogRegisterBufData(0, &dead_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, dead,
- sizeof(OffsetNumber) * ndead);
+ meta_pack_sz = log_heap_prune_and_freeze_pack_meta(
+ &unused_meta, unused_meta_pack);
+
+ XLogRegisterBufData(0, &unused_meta_pack, meta_pack_sz);
+ XLogRegisterBufData(0, unused_meta.pack, unused_meta.nbytes);
+ }
+ }
+ else
+ {
+ /* Otherwise, we can't use DFoR compression */
+ xlrec.flags &= ~XLHP_DFOR_COMPRESSED;
+ }
}
- if (nunused > 0)
+
+ if ((xlrec.flags & XLHP_DFOR_COMPRESSED) == 0)
{
- xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+ if (nfrozen > 0)
+ XLogRegisterBufData(0, frz_offsets,
+ sizeof(OffsetNumber) * nfrozen);
+ if (nredirected > 0)
+ {
+ xlrec.flags |= XLHP_HAS_REDIRECTIONS;
- unused_items.ntargets = nunused;
- XLogRegisterBufData(0, &unused_items,
- offsetof(xlhp_prune_items, data));
- XLogRegisterBufData(0, unused,
- sizeof(OffsetNumber) * nunused);
+ redirect_items.ntargets = nredirected;
+ XLogRegisterBufData(0, &redirect_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, redirected,
+ sizeof(OffsetNumber[2]) * nredirected);
+ }
+ if (ndead > 0)
+ {
+ xlrec.flags |= XLHP_HAS_DEAD_ITEMS;
+
+ dead_items.ntargets = ndead;
+ XLogRegisterBufData(0, &dead_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, dead, sizeof(OffsetNumber) * ndead);
+ }
+ if (nunused > 0)
+ {
+ xlrec.flags |= XLHP_HAS_NOW_UNUSED_ITEMS;
+
+ unused_items.ntargets = nunused;
+ XLogRegisterBufData(0, &unused_items,
+ offsetof(xlhp_prune_items, data));
+ XLogRegisterBufData(0, unused, sizeof(OffsetNumber) * nunused);
+ }
}
- if (nfrozen > 0)
- XLogRegisterBufData(0, frz_offsets,
- sizeof(OffsetNumber) * nfrozen);
/*
* Prepare the main xl_heap_prune record. We already set the XLHP_HAS_*
diff --git a/src/backend/access/rmgrdesc/Makefile b/src/backend/access/rmgrdesc/Makefile
index cd95eec37f1..49e9c46145f 100644
--- a/src/backend/access/rmgrdesc/Makefile
+++ b/src/backend/access/rmgrdesc/Makefile
@@ -17,6 +17,7 @@ OBJS = \
gindesc.o \
gistdesc.o \
hashdesc.o \
+ heapam_xlog_dfor.o \
heapdesc.o \
logicalmsgdesc.o \
mxactdesc.o \
diff --git a/src/backend/access/rmgrdesc/heapam_xlog_dfor.c b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
new file mode 100644
index 00000000000..bc38e4a2555
--- /dev/null
+++ b/src/backend/access/rmgrdesc/heapam_xlog_dfor.c
@@ -0,0 +1,114 @@
+#include "lib/bitpack_u16.h"
+#include "access/heapam_xlog_dfor.h"
+
+bool wal_prune_dfor_compression = true;
+
+size_t
+log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta, uint8 buf[])
+{
+ size_t caret = 0;
+ caret = bitpack_u16_pack(buf, caret, meta->item_cnt,
+ XLHPF_META_ITEM_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->delta_wid,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->signed_deltas == 0) ? 0 : 1,
+ XLHPF_META_SIGNEDDELTAS_FLAG_SZ);
+ caret = bitpack_u16_pack(buf, caret, (meta->exc_cnt == 0) ? 0 : 1,
+ XLHPF_META_EXCEPTION_FLAG_SZ);
+ if (meta->exc_cnt != 0)
+ {
+ caret = bitpack_u16_pack(buf, caret, meta->exc_cnt,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_wid,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ caret = bitpack_u16_pack(buf, caret, meta->exc_pos_wid,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ dfor_meta_t checker;
+ log_heap_prune_and_freeze_unpack_meta(&checker, buf);
+ Assert(meta->item_cnt == checker.item_cnt);
+ Assert(meta->delta_wid == checker.delta_wid);
+ Assert(meta->signed_deltas == checker.signed_deltas);
+ Assert(meta->exc_cnt == checker.exc_cnt);
+ Assert(meta->exc_wid == checker.exc_wid);
+ Assert(meta->exc_pos_wid == checker.exc_pos_wid);
+ }
+#endif
+ return (caret + 7) / 8; /* the length of packed dfor_meta, in bytes*/
+}
+
+size_t
+log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[])
+{
+ size_t caret = 0;
+ bool exc;
+
+ meta->item_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_ITEM_COUNT_SZ);
+ meta->delta_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_DELTA_WIDTH_SZ);
+ meta->signed_deltas = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_SIGNEDDELTAS_FLAG_SZ);
+ exc = bitpack_u16_unpack(packed_meta, &caret, XLHPF_META_EXCEPTION_FLAG_SZ);
+
+ if (exc)
+ {
+ meta->exc_cnt = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_COUNT_SZ);
+ meta->exc_wid = bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_WIDTH_SZ);
+ meta->exc_pos_wid =
+ bitpack_u16_unpack(packed_meta, &caret,
+ XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ);
+ }
+ else
+ {
+ meta->exc_cnt = 0;
+ meta->exc_wid = 0;
+ meta->exc_pos_wid = 0;
+ }
+ meta->nbytes = dfor_u16_calc_nbytes(*meta);
+
+ Assert(meta->item_cnt != 0);
+ Assert(meta->delta_wid != 0);
+ Assert(meta->nbytes != 0);
+
+ return (caret + 7) / 8;
+}
+
+void
+heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items,
+ uint8 dfor_buf[])
+{
+ dfor_meta_t dfor = {0};
+ size_t packed_meta_nbytes;
+ uniqsortvect_u16_t vect;
+
+ packed_meta_nbytes =
+ log_heap_prune_and_freeze_unpack_meta(&dfor, (uint8*) *cursor);
+
+ *cursor += packed_meta_nbytes;
+
+ dfor.pack = (uint8 *)*cursor;
+ dfor_u16_unpack(&dfor, &vect, 4 * DFOR_BUF_PART_SIZE,
+ dfor_buf);
+
+ *cursor += dfor.nbytes;
+
+ Assert(dfor.nbytes != 0);
+
+ Assert(vect.cnt != 0);
+ Assert(vect.mem_is_outer == true);
+ Assert((void*)vect.m == (void*)dfor_buf);
+
+ *nitems = vect.cnt;
+ *items = vect.m;
+}
diff --git a/src/backend/access/rmgrdesc/heapdesc.c b/src/backend/access/rmgrdesc/heapdesc.c
index 75ae6f9d375..a445b21ed36 100644
--- a/src/backend/access/rmgrdesc/heapdesc.c
+++ b/src/backend/access/rmgrdesc/heapdesc.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/heapam_xlog.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/rmgrdesc_utils.h"
#include "access/visibilitymapdefs.h"
#include "storage/standbydefs.h"
@@ -102,17 +103,19 @@ plan_elem_desc(StringInfo buf, void *plan, void *data)
* This is in heapdesc.c so it can be shared between heap2_redo and heap2_desc
* code, the latter of which is used in frontend (pg_waldump) code.
*/
-void
+char *
heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused)
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[])
{
if (flags & XLHP_HAS_FREEZE_PLANS)
{
- xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *) cursor;
+ int nfrozen = 0;
+ xlhp_freeze_plans *freeze_plans = (xlhp_freeze_plans *)cursor;
*nplans = freeze_plans->nplans;
Assert(*nplans > 0);
@@ -120,6 +123,20 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
cursor += offsetof(xlhp_freeze_plans, plans);
cursor += sizeof(xlhp_freeze_plan) * *nplans;
+
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, &nfrozen, frz_offsets,
+ dfor_buf);
+ }
+ else
+ {
+ for (int i = 0; i < *nplans; i++)
+ nfrozen += (*plans)[i].ntuples;
+
+ *frz_offsets = (OffsetNumber *) cursor;
+ cursor += sizeof(OffsetNumber) * nfrozen;
+ }
}
else
{
@@ -129,14 +146,25 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_REDIRECTIONS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, nredirected, redirected,
+ dfor_buf + DFOR_BUF_PART_SIZE);
+ Assert(*nredirected % 2 == 0);
+ *nredirected /= 2;
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+
+ *nredirected = subrecord->ntargets;
+ Assert(*nredirected > 0);
+ *redirected = &subrecord->data[0];
- *nredirected = subrecord->ntargets;
- Assert(*nredirected > 0);
- *redirected = &subrecord->data[0];
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber[2]) * *nredirected;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber[2]) * *nredirected;
+ }
}
else
{
@@ -146,14 +174,22 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_DEAD_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if(flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, ndead, nowdead,
+ dfor_buf + 2 * DFOR_BUF_PART_SIZE);
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *ndead = subrecord->ntargets;
- Assert(*ndead > 0);
- *nowdead = subrecord->data;
+ *ndead = subrecord->ntargets;
+ Assert(*ndead > 0);
+ *nowdead = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *ndead;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *ndead;
+ }
}
else
{
@@ -163,22 +199,29 @@ heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
if (flags & XLHP_HAS_NOW_UNUSED_ITEMS)
{
- xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
+ if (flags & XLHP_DFOR_COMPRESSED)
+ {
+ heap_xlog_deserialize_dfor(&cursor, nunused, nowunused,
+ dfor_buf + 3 * DFOR_BUF_PART_SIZE);
+ }
+ else
+ {
+ xlhp_prune_items *subrecord = (xlhp_prune_items *) cursor;
- *nunused = subrecord->ntargets;
- Assert(*nunused > 0);
- *nowunused = subrecord->data;
+ *nunused = subrecord->ntargets;
+ Assert(*nunused > 0);
+ *nowunused = subrecord->data;
- cursor += offsetof(xlhp_prune_items, data);
- cursor += sizeof(OffsetNumber) * *nunused;
+ cursor += offsetof(xlhp_prune_items, data);
+ cursor += sizeof(OffsetNumber) * *nunused;
+ }
}
else
{
*nunused = 0;
*nowunused = NULL;
}
-
- *frz_offsets = (OffsetNumber *) cursor;
+ return cursor;
}
void
@@ -309,13 +352,16 @@ heap2_desc(StringInfo buf, XLogReaderState *record)
xlhp_freeze_plan *plans;
OffsetNumber *frz_offsets;
+ uint8 dfor_buf[5 * DFOR_BUF_PART_SIZE];
+
char *cursor = XLogRecGetBlockData(record, 0, &datalen);
heap_xlog_deserialize_prune_and_freeze(cursor, xlrec->flags,
&nplans, &plans, &frz_offsets,
&nredirected, &redirected,
&ndead, &nowdead,
- &nunused, &nowunused);
+ &nunused, &nowunused,
+ dfor_buf);
appendStringInfo(buf, ", nplans: %u, nredirected: %u, ndead: %u, nunused: %u",
nplans, nredirected, ndead, nunused);
diff --git a/src/backend/access/rmgrdesc/meson.build b/src/backend/access/rmgrdesc/meson.build
index d9000ccd9fd..6ceea4514ec 100644
--- a/src/backend/access/rmgrdesc/meson.build
+++ b/src/backend/access/rmgrdesc/meson.build
@@ -11,6 +11,7 @@ rmgr_desc_sources = files(
'gistdesc.c',
'hashdesc.c',
'heapdesc.c',
+ 'heapam_xlog_dfor.c',
'logicalmsgdesc.c',
'mxactdesc.c',
'nbtdesc.c',
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index 83af594d4af..c53e2921c01 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3507,6 +3507,13 @@
boot_val => 'false',
},
+{ name => 'wal_prune_dfor_compression', type => 'bool', context => 'PGC_SUSET', group => 'WAL_SETTINGS',
+ short_desc => 'Compress dead and unused offset arrays at PRUNE/FREEZE WAL records using DFOR.',
+ long_desc => 'Enables compression of dead and unused OffsetNumber arrays stored in heap PRUNE/FREEZE WAL records using customised delta frame-of-reference encoding.',
+ variable => 'wal_prune_dfor_compression',
+ boot_val => 'true'
+},
+
{ name => 'wal_receiver_create_temp_slot', type => 'bool', context => 'PGC_SIGHUP', group => 'REPLICATION_STANDBY',
short_desc => 'Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured.',
variable => 'wal_receiver_create_temp_slot',
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 290ccbc543e..bee60b2ebcd 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -31,6 +31,7 @@
#include "access/commit_ts.h"
#include "access/gin.h"
+#include "access/heapam_xlog_dfor.h"
#include "access/slru.h"
#include "access/toast_compression.h"
#include "access/twophase.h"
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ac38cddaaf9..4152e789ee1 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -263,6 +263,8 @@
# (change requires restart)
#wal_compression = off # enables compression of full-page writes;
# off, pglz, lz4, zstd, or on
+#wal_prune_dfor_compression = true # Compress dead and unused offset arrays
+ # at PRUNE/FREEZE WAL records using DFOR.
#wal_init_zero = on # zero-fill new WAL files
#wal_recycle = on # recycle WAL files
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
diff --git a/src/bin/pg_waldump/.gitignore b/src/bin/pg_waldump/.gitignore
index ec51f41c767..a3c02446b9d 100644
--- a/src/bin/pg_waldump/.gitignore
+++ b/src/bin/pg_waldump/.gitignore
@@ -10,6 +10,7 @@
/gistdesc.c
/hashdesc.c
/heapdesc.c
+/heapam_xlog_dfor.c
/logicalmsgdesc.c
/mxactdesc.c
/nbtdesc.c
@@ -28,5 +29,13 @@
/xlogreader.c
/xlogstats.c
+# Source files copied from src/backend/lib
+/bitpack_templ.c
+/bitpack_u16.c
+/dfor_templ.c
+/dfor_u16.c
+/vect_templ.c
+/vect_u16.c
+
# Generated by test suite
/tmp_check/
diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile
index aabb87566a2..5e521c1e822 100644
--- a/src/bin/pg_waldump/Makefile
+++ b/src/bin/pg_waldump/Makefile
@@ -8,8 +8,9 @@ export TAR
subdir = src/bin/pg_waldump
top_builddir = ../../..
-include $(top_builddir)/src/Makefile.global
+dfor_dir := $(top_builddir)/src/backend/lib
+include $(top_builddir)/src/Makefile.global
OBJS = \
$(RMGRDESCOBJS) \
$(WIN32RES) \
@@ -20,10 +21,13 @@ OBJS = \
xlogreader.o \
xlogstats.o
+include $(dfor_dir)/Makefile.dfor
+OBJS += $(OBJS_DFOR)
+
override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
-RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c)))
+RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc*.c))) heapam_xlog_dfor.c
RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES))
@@ -32,6 +36,24 @@ all: pg_waldump
pg_waldump: $(OBJS) | submake-libpgport
$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+bitpack_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+bitpack_u16.c: % : $(top_srcdir)/src/backend/lib/% bitpack_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+dfor_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+dfor_u16.c: % : $(top_srcdir)/src/backend/lib/% dfor_templ.c
+ rm -f $@ && $(LN_S) $< .
+
+vect_templ.c: % : $(top_srcdir)/src/backend/lib/%
+ rm -f $@ && $(LN_S) $< .
+
+vect_u16.c: % : $(top_srcdir)/src/backend/lib/% vect_templ.c
+ rm -f $@ && $(LN_S) $< .
+
xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/%
rm -f $@ && $(LN_S) $< .
diff --git a/src/bin/pg_waldump/meson.build b/src/bin/pg_waldump/meson.build
index 5296f21b82c..c33be88712c 100644
--- a/src/bin/pg_waldump/meson.build
+++ b/src/bin/pg_waldump/meson.build
@@ -10,6 +10,7 @@ pg_waldump_sources = files(
pg_waldump_sources += rmgr_desc_sources
pg_waldump_sources += xlogreader_sources
pg_waldump_sources += files('../../backend/access/transam/xlogstats.c')
+pg_waldump_sources += dfor_sources
if host_system == 'windows'
pg_waldump_sources += rc_bin_gen.process(win32ver_rc, extra_args: [
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index fdca7d821c8..2233ff7d108 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -15,6 +15,7 @@
#define HEAPAM_XLOG_H
#include "access/htup.h"
+#include "access/htup_details.h"
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
#include "storage/buf.h"
@@ -341,6 +342,8 @@ typedef struct xl_heap_prune
#define XLHP_VM_ALL_VISIBLE (1 << 8)
#define XLHP_VM_ALL_FROZEN (1 << 9)
+#define XLHP_DFOR_COMPRESSED (1 << 10)
+
/*
* xlhp_freeze_plan describes how to freeze a group of one or more heap tuples
* (appears in xl_heap_prune's xlhp_freeze_plans sub-record)
@@ -489,11 +492,12 @@ extern const char *heap2_identify(uint8 info);
extern void heap_xlog_logical_rewrite(XLogReaderState *r);
/* in heapdesc.c, so it can be shared between frontend/backend code */
-extern void heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
+extern char * heap_xlog_deserialize_prune_and_freeze(char *cursor, uint16 flags,
int *nplans, xlhp_freeze_plan **plans,
OffsetNumber **frz_offsets,
int *nredirected, OffsetNumber **redirected,
int *ndead, OffsetNumber **nowdead,
- int *nunused, OffsetNumber **nowunused);
+ int *nunused, OffsetNumber **nowunused,
+ uint8 dfor_buf[]);
-#endif /* HEAPAM_XLOG_H */
+#endif /* HEAPAM_XLOG_H */
\ No newline at end of file
diff --git a/src/include/access/heapam_xlog_dfor.h b/src/include/access/heapam_xlog_dfor.h
new file mode 100644
index 00000000000..b169696e9bc
--- /dev/null
+++ b/src/include/access/heapam_xlog_dfor.h
@@ -0,0 +1,144 @@
+#ifndef HEAPAM_XLOG_DFOR_H
+#define HEAPAM_XLOG_DFOR_H
+
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/dfor_u16.h"
+#include "storage/bufpage.h"
+
+/*
+ * DFoR's meta block for PRUNE/FREEZE record
+ *
+ * A meta block contains parameters required for decompression of the following
+ * DFoR pack. It is densely bit-packed. If the exception flag is zero, fields
+ * pertaining to exceptions is absent, which means that DFoR pack does not
+ * contain exceptions. Calculation of field widths takes into account
+ * next considerations:
+ *
+ * Max Item Count should be more or equal to MaxHeapTuplesPerPage * 2. We
+ * multiply it by 2 since the sequence of redirected tuple offsets have a pair
+ * of offsets for each redirection and in the worst case we can have all tuples
+ * redirected.
+ *
+ * Since we can't calculate MaxHeapTuplesPerPage on preprocessor
+ * stage, we intentionally overestimate it as:
+ * Max Item Count > BLCKSZ / Min Tuple Size = BLCKSZ / 24
+ * to provide a margin. In general, depending on BLCKSZ, it should not result in
+ * DFoR meta block overhead.
+ * For instance, for a block size of 32768, we have Max Item Count on a page = 1366. Redirected sequences can and
+ * it needs 11 bits width field.
+ *
+ * Size of field Item Count:
+ * ITEM_COUNT_SZ = log2(MaxItemCount) * 2.
+ * We mu
+ *
+ * Maximum Delta Width is equal to ITEM_COUNT_SZ. So DELTA_WIDTH_SZ in a DFoR
+ * meta block can be calculated as:
+ * DELTA_WIDTH_SZ >= log2(Max Delta Width) = log2(ITEM_COUNT_SZ)
+ *
+ * Max Exception Count = 0.1 * MaxItemCount, according to DFoR algorithm which
+ * guarantees that not less than 90% of items will be covered without using
+ * exceptions. So:
+ * EXCEPTION_COUNT_SZ >= log2(0.1 * MaxItemCount).
+ *
+ * An exception is part of a delta, exceeding choosen delta width. Exception is
+ * saved in separated part of DFoR pack and, since delta width is not less
+ * than 1:
+ * EXCEPTION_WIDTH_SZ >= log2(Max Delta Width - 1) = log2(ITEM_COUNT_SZ - 1).
+ *
+ * An exception's position shows the position of of a delta to wich the
+ * exception has to be applied. Values of an exception position must cover the
+ * same value range as an Item Count, so the Max Width of an Exception Position
+ * is equal to width of Delta. Consequently, the size of Exception Position
+ * Width calculated as:
+ * EXCEPTION_POSITION_WIDTH_SIZE = log2(Max Delta Width) = DELTA_WIDTH_SZ
+ *
+ * For example, Meta for BLCKSZ equal to 32768 has next sizes of field
+ * | sect. | byte | bits | param | size | range of values |
+ * |-------|------|-----------|--------------------|--------|-----------------|
+ * | main | 0, 1 | 0-11 | item count | 12 bit | 1...2730 |
+ * | main | 1 | 12-15 | delta width | 4 bit | 1...12 |
+ * | main | 2 | 16 | signed deltas flag | 1 bit | 0...1 |
+ * | main | 2 | 17 | exception flag | 1 bit | 0...1 |
+ * | extra | 2, 3 | 18-26 | exception count | 9 bit | 1...274 |
+ * | extra | 3 | 27-29 | exception width | 4 bit | 1...11 |
+ * | extra | 3, 4 | 30-33 | except pos. width | 4 bit | 1...12 |
+ */
+
+/*
+ * The sizes of fields in the compressed DFoR Meta structure of an
+ * XLOG_HEAP2_PRUNE* record.
+ */
+#if BLCKSZ == 32768
+#define XLHPF_META_ITEM_COUNT_SZ 12
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 16384
+#define XLHPF_META_ITEM_COUNT_SZ 11
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 8192
+#define XLHPF_META_ITEM_COUNT_SZ 10
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 4096
+#define XLHPF_META_ITEM_COUNT_SZ 9
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 2048
+#define XLHPF_META_ITEM_COUNT_SZ 8
+#define XLHPF_META_DELTA_WIDTH_SZ 4
+#elif BLCKSZ == 1024
+#define XLHPF_META_ITEM_COUNT_SZ 7
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 512
+#define XLHPF_META_ITEM_COUNT_SZ 6
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 256
+#define XLHPF_META_ITEM_COUNT_SZ 5
+#define XLHPF_META_DELTA_WIDTH_SZ 3
+#elif BLCKSZ == 128
+#define XLHPF_META_ITEM_COUNT_SZ 4
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#elif BLCKSZ == 64
+#define XLHPF_META_ITEM_COUNT_SZ 3
+#define XLHPF_META_DELTA_WIDTH_SZ 2
+#else
+#error "Unsupported BLCKSZ in XLog Heap And Prune."
+#endif
+
+#define XLHPF_META_SIGNEDDELTAS_FLAG_SZ 1 /* Flag about signedness of deltas */
+
+#define XLHPF_META_EXCEPTION_FLAG_SZ 1 /* Flag about Extra Section presence */
+
+/* Size of Exception Count field */
+#if XLHPF_META_ITEM_COUNT_SZ > 6
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ - 3
+#else
+#define XLHPF_META_EXCEPTION_COUNT_SZ XLHPF_META_ITEM_COUNT_SZ 3
+#endif
+
+#define XLHPF_META_EXCEPTION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Width field */
+
+#define XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ \
+ XLHPF_META_DELTA_WIDTH_SZ /* Size of Exception Position width field */
+
+/* Maximal size of packed meta */
+#define MAX_PACKED_META_SIZE \
+ (XLHPF_META_ITEM_COUNT_SZ + XLHPF_META_DELTA_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_FLAG_SZ + XLHPF_META_EXCEPTION_POSITION_WIDTH_SZ + \
+ XLHPF_META_EXCEPTION_WIDTH_SZ + XLHPF_META_EXCEPTION_COUNT_SZ + 7) / 8
+
+/* The size of a typical chunk of memory used by dfor_pack */
+#define DFOR_BUF_PART_SIZE MaxHeapTuplesPerPage * sizeof(uint16)
+
+extern bool wal_prune_dfor_compression; /* GUC */
+
+extern size_t log_heap_prune_and_freeze_pack_meta(const dfor_meta_t *meta,
+ uint8 buf[]);
+
+extern size_t log_heap_prune_and_freeze_unpack_meta(dfor_meta_t *meta,
+ const uint8 packed_meta[]);
+
+extern void heap_xlog_deserialize_dfor(char **cursor, int *nitems,
+ OffsetNumber **items, uint8 dfor_buf[]);
+
+#endif /* HEAPAM_XLOG_DFOR_H */
\ No newline at end of file
diff --git a/src/test/dfor/Makefile b/src/test/dfor/Makefile
index 2ca98f76a0f..f7a287a1d3b 100644
--- a/src/test/dfor/Makefile
+++ b/src/test/dfor/Makefile
@@ -48,6 +48,8 @@ check-unit: $(TESTS)
cd $(top_builddir)/$(subdir) && \
$(PROVE) $(PROVE_FLAGS) \
-v $(addprefix ./,$(if $(PROVE_TESTS), $(PROVE_TESTS), $(TESTS)))
+# The example of using the check-unit rule:
+# make check-unit PROVE_TESTS='test_dfor_u16' PROVE_FLAGS='--verbose'
check: check-unit
diff --git a/src/test/recovery/t/052_prune_dfor_compression.pl b/src/test/recovery/t/052_prune_dfor_compression.pl
new file mode 100644
index 00000000000..951478fbbd3
--- /dev/null
+++ b/src/test/recovery/t/052_prune_dfor_compression.pl
@@ -0,0 +1,283 @@
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# ------------------------------------------------------------
+# Workload generating dead tuples and PRUNE WAL
+# ------------------------------------------------------------
+sub generate_prune_workload
+{
+ my ($node, $workload) = @_;
+
+
+ my $start_lsn;
+ my $end_lsn;
+
+ if ($workload eq "vacuum_with_index"
+ || $workload eq "vacuum_no_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE TABLE t_prune (
+ id int,
+ val text
+ ) WITH (fillfactor = 100, autovacuum_enabled = false);
+ });
+
+ $node->safe_psql('postgres', q{
+ SET vacuum_freeze_min_age = 0;
+ SET vacuum_freeze_table_age = 0;
+ });
+
+ # -------------------------
+ # Phase 1: INSERT
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ INSERT INTO t_prune
+ SELECT g, 'x'
+ FROM generate_series(1,3000000) g;
+ });
+
+ # Optional index
+ if ($workload eq "vacuum_with_index")
+ {
+ $node->safe_psql('postgres', q{
+ CREATE INDEX ON t_prune(id);
+ });
+ }
+ # -------------------------
+ # Phase 2: DELETE + VACUUM
+ # -------------------------
+ $node->safe_psql('postgres', q{
+ DELETE FROM t_prune
+ WHERE id % 500 <> 0;
+ });
+
+ # Force WAL flush and capture LSN
+ $start_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+
+ # VACUUM cycles to trigger PRUNE
+ for my $i (1..3)
+ {
+ $node->safe_psql('postgres', q{ VACUUM FREEZE t_prune; });
+ }
+
+ $end_lsn = $node->safe_psql('postgres', q{
+ SELECT pg_current_wal_flush_lsn();
+ });
+ }
+ else
+ {
+ die "Workload is not defined: workload=$workload";
+ }
+
+ chomp($start_lsn);
+ print "Captured start LSN: $start_lsn\n";
+ chomp($end_lsn);
+ print "Captured end LSN: $end_lsn\n";
+
+ return ($start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# WAL analyzer
+# ------------------------------------------------------------
+sub collect_wal_stats
+{
+ my ($node, $start_lsn, $end_lsn) = @_;
+
+ my $wal_dir = $node->data_dir . "/pg_wal";
+
+ print "wal_dir=" . $wal_dir . "\n";
+ print "collect_wal_stats: start_lsn=$start_lsn\n";
+ print "collect_wal_stats: end_lsn=$end_lsn\n";
+
+ my $cmd;
+
+ if (defined $end_lsn && $end_lsn ne '')
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn -e $end_lsn 2>/dev/null";
+ }
+ else
+ {
+ $cmd = "pg_waldump -p $wal_dir -s $start_lsn 2>/dev/null";
+ }
+
+ my @lines = `$cmd`;
+
+ # -------------------------
+ # Counters
+ # -------------------------
+ my $total_records = 0;
+ my $total_bytes = 0;
+
+ my $prune_records = 0;
+ my $prune_bytes = 0;
+
+ foreach my $line (@lines)
+ {
+ # Extract total record size
+ if ($line =~ /len \(rec\/tot\):\s*\d+\/\s*(\d+)/)
+ {
+ my $size = $1;
+
+ $total_records++;
+ $total_bytes += $size;
+
+ # PRUNE-specific tracking
+ if ($line =~ /PRUNE_VACUUM_SCAN/)
+ {
+ $prune_records++;
+ $prune_bytes += $size;
+ }
+ }
+ }
+
+ if ($total_records == 0)
+ {
+ die "No WAL records found in range $start_lsn → $end_lsn";
+ }
+
+ print "TOTAL: records=$total_records; bytes=$total_bytes\n";
+ print "PRUNE: records=$prune_records; bytes=$prune_bytes\n";
+
+ return {
+ total_records => $total_records,
+ total_bytes => $total_bytes,
+ prune_records => $prune_records,
+ prune_bytes => $prune_bytes,
+ };
+}
+
+# ------------------------------------------------------------
+# Run test on a fresh cluster
+# ------------------------------------------------------------
+sub run_cluster_test
+{
+ my ($name, $compression, $workload) = @_;
+
+ my $node = PostgreSQL::Test::Cluster->new($name);
+
+ $node->init;
+
+ $node->append_conf('postgresql.conf', qq{
+ wal_level = replica
+ autovacuum = off
+ wal_prune_dfor_compression = $compression
+ wal_keep_size = '1GB'
+ max_wal_size = '20GB'
+ });
+
+ $node->start;
+
+ my ($start_lsn, $end_lsn) = generate_prune_workload($node, $workload);
+
+ $node->stop;
+
+ return collect_wal_stats($node, $start_lsn, $end_lsn);
+}
+
+# ------------------------------------------------------------
+# Formatting helpers
+# ------------------------------------------------------------
+
+sub _pct_reduction
+{
+ my ($before, $after) = @_;
+ return "N/A" if $before == 0;
+
+ my $pct = 100 * ($before - $after) / $before;
+ return sprintf("%d%%", int($pct + 0.5));
+}
+
+sub _ratio
+{
+ my ($before, $after) = @_;
+ return "N/A" if $after == 0;
+
+ my $r = $before / $after;
+ return sprintf("%.1fx", $r);
+}
+
+# ------------------------------------------------------------
+# Report: total WAL stats
+# ------------------------------------------------------------
+sub report_wal_diff
+{
+ my ($off, $on) = @_;
+
+ my $b_bytes = $off->{total_bytes};
+ my $a_bytes = $on->{total_bytes};
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17s %17s %11s\n",
+ "", "DFOR off, bytes", "DFOR on, bytes", "Reduction";
+
+ printf "%-20s %17s %17s %11s\n",
+ "-" x 20, "-" x 17, "-" x 17, "-" x 11;
+
+ printf "%-20s %17d %17d %11s\n",
+ "WAL total size",
+ $off->{total_bytes},
+ $on->{total_bytes},
+ _pct_reduction($off->{total_bytes}, $on->{total_bytes});
+
+ printf "%-20s %17d %17d %11s\n\n",
+ "Prune records size",
+ $off->{prune_bytes},
+ $on->{prune_bytes},
+ _ratio($off->{prune_bytes}, $on->{prune_bytes});
+}
+
+# ------------------------------------------------------------
+# Scenario 1: VACUUM without index
+# ------------------------------------------------------------
+my $off_noidx = run_cluster_test("prune_off_noidx", "off", "vacuum_no_index");
+my $on_noidx = run_cluster_test("prune_on_noidx", "on", "vacuum_no_index");
+
+cmp_ok(
+ $off_noidx->{prune_bytes},
+ '>',
+ $on_noidx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having no index.'
+);
+
+cmp_ok(
+ $off_noidx->{total_bytes},
+ '>',
+ $on_noidx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having no index.'
+);
+
+print "\n\n=== VACUUM (table with no index) ===\n";
+report_wal_diff($off_noidx, $on_noidx);
+
+# ------------------------------------------------------------
+# Scenario 2: VACUUM with index
+# ------------------------------------------------------------
+my $off_idx = run_cluster_test("prune_off_idx", "off", "vacuum_with_index");
+my $on_idx = run_cluster_test("prune_on_idx", "on", "vacuum_with_index");
+
+cmp_ok(
+ $off_idx->{prune_bytes},
+ '>',
+ $on_idx->{prune_bytes},
+ 'DFOR reduces the PRUNE WAL size on vacuuming a table having an index.'
+);
+
+cmp_ok(
+ $off_idx->{total_bytes},
+ '>=',
+ $on_idx->{total_bytes},
+ 'DFOR reduces the total WAL size on vacuuming a table having an index.'
+);
+
+print "\n\n=== VACUUM (table with index) ===\n";
+report_wal_diff($off_idx, $on_idx);
+
+done_testing();
--
2.53.0
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2026-04-21 08:48 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-03-07 13:56 Compress prune/freeze records with Delta Frame of Reference algorithm Evgeny Voropaev <[email protected]>
2026-03-20 10:05 ` Evgeny Voropaev <[email protected]>
2026-03-22 16:20 ` Andres Freund <[email protected]>
2026-03-24 14:28 ` Evgeny Voropaev <[email protected]>
2026-03-24 16:25 ` Evgeny Voropaev <[email protected]>
2026-04-08 12:34 ` Evgeny Voropaev <[email protected]>
2026-04-09 04:21 ` Evgeny Voropaev <[email protected]>
2026-04-09 10:00 ` Evgeny Voropaev <[email protected]>
2026-04-09 16:50 ` Andres Freund <[email protected]>
2026-04-09 17:55 ` Andrey Borodin <[email protected]>
2026-04-10 08:44 ` Evgeny Voropaev <[email protected]>
2026-04-14 07:08 ` Evgeny Voropaev <[email protected]>
2026-04-14 09:02 ` Heikki Linnakangas <[email protected]>
2026-04-14 09:11 ` Andrey Borodin <[email protected]>
2026-04-21 05:41 ` Evgeny Voropaev <[email protected]>
2026-04-21 07:20 ` Heikki Linnakangas <[email protected]>
2026-04-21 08:48 ` Evgeny Voropaev <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox