agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
20+ messages / 6 participants
[nested] [flat]
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8a2c6fc122..daca2e0085 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -672,11 +672,11 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Wed_Jan_30_10_42_04_2019_681)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v12-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test.
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
It is convenient that priary_slot_name can be specified on taking a
base backup. This adds a new parameter of the name to the perl
function.
---
src/test/perl/PostgresNode.pm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index 8a2c6fc122..daca2e0085 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -672,11 +672,11 @@ sub init_from_backup
chmod(0700, $data_path);
# Base configuration for this node
- $self->append_conf(
- 'postgresql.conf',
- qq(
-port = $port
-));
+ $self->append_conf('postgresql.conf', qq(port = $port));
+ $self->append_conf('postgresql.conf',
+ qq(primary_slot_name = $params{primary_slot_name}))
+ if (defined $params{primary_slot_name});
+
$self->enable_streaming($root_node) if $params{has_streaming};
$self->enable_restoring($root_node) if $params{has_restoring};
return;
--
2.16.3
----Next_Part(Thu_Dec_20_16_24_38_2018_792)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v11-0004-TAP-test-for-the-slot-limit-feature.patch"
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH 1/2] Add IntegerSet, to hold large sets of 64-bit ints efficiently.
@ 2019-03-20 00:26 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Heikki Linnakangas @ 2019-03-20 00:26 UTC (permalink / raw)
The set is implemented as a B-tree, with a compact representation at leaf
items, using Simple-8b algorithm, so that clusters of nearby values take
less space.
This doesn't include any use of the code yet, but we have two patches in
the works that would benefit from this:
* the GiST vacuum patch, to track empty GiST pages and internal GiST pages.
* Reducing memory usage, and also allowing more than 1 GB of memory to be
used, to hold the dead TIDs in VACUUM.
This includes a unit test module, in src/test/modules/test_integerset.
It can be used to verify correctness, as a regression test, but if you run
it manully, it can also print memory usage and execution time of some of
the tests.
Author: Heikki Linnakangas, Andrey Borodin
Discussion: https://www.postgresql.org/message-id/[email protected]
---
src/backend/lib/Makefile | 2 +-
src/backend/lib/README | 2 +
src/backend/lib/integerset.c | 1039 +++++++++++++++++
src/include/lib/integerset.h | 25 +
src/test/modules/Makefile | 1 +
src/test/modules/test_integerset/.gitignore | 4 +
src/test/modules/test_integerset/Makefile | 21 +
src/test/modules/test_integerset/README | 7 +
.../expected/test_integerset.out | 14 +
.../test_integerset/sql/test_integerset.sql | 11 +
.../test_integerset/test_integerset--1.0.sql | 8 +
.../modules/test_integerset/test_integerset.c | 622 ++++++++++
.../test_integerset/test_integerset.control | 4 +
13 files changed, 1759 insertions(+), 1 deletion(-)
create mode 100644 src/backend/lib/integerset.c
create mode 100644 src/include/lib/integerset.h
create mode 100644 src/test/modules/test_integerset/.gitignore
create mode 100644 src/test/modules/test_integerset/Makefile
create mode 100644 src/test/modules/test_integerset/README
create mode 100644 src/test/modules/test_integerset/expected/test_integerset.out
create mode 100644 src/test/modules/test_integerset/sql/test_integerset.sql
create mode 100644 src/test/modules/test_integerset/test_integerset--1.0.sql
create mode 100644 src/test/modules/test_integerset/test_integerset.c
create mode 100644 src/test/modules/test_integerset/test_integerset.control
diff --git a/src/backend/lib/Makefile b/src/backend/lib/Makefile
index 191ea9bca26..3c1ee1df83a 100644
--- a/src/backend/lib/Makefile
+++ b/src/backend/lib/Makefile
@@ -13,6 +13,6 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = binaryheap.o bipartite_match.o bloomfilter.o dshash.o hyperloglog.o \
- ilist.o knapsack.o pairingheap.o rbtree.o stringinfo.o
+ ilist.o integerset.o knapsack.o pairingheap.o rbtree.o stringinfo.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/lib/README b/src/backend/lib/README
index ae5debe1bc6..f2fb591237d 100644
--- a/src/backend/lib/README
+++ b/src/backend/lib/README
@@ -13,6 +13,8 @@ hyperloglog.c - a streaming cardinality estimator
ilist.c - single and double-linked lists
+integerset.c - a data structure for holding large set of integers
+
knapsack.c - knapsack problem solver
pairingheap.c - a pairing heap
diff --git a/src/backend/lib/integerset.c b/src/backend/lib/integerset.c
new file mode 100644
index 00000000000..c9172fa2005
--- /dev/null
+++ b/src/backend/lib/integerset.c
@@ -0,0 +1,1039 @@
+/*-------------------------------------------------------------------------
+ *
+ * integerset.c
+ * Data structure to hold a large set of 64-bit integers efficiently
+ *
+ * IntegerSet provides an in-memory data structure to hold a set of
+ * arbitrary 64-bit integers. Internally, the values are stored in a
+ * B-tree, with a special packed representation at the leaf level using
+ * the Simple-8b algorithm, which can pack hold clusters of nearby values
+ * very tightly.
+ *
+ * Memory consumption depends on the number of values stored, but also
+ * on how far the values are from each other. In the best case, with
+ * long runs of consecutive integers, memory consumption can be as low as
+ * 0.1 bytes per integer. In the worst case, if integers are more than
+ * 2^32 apart, it uses about 8 bytes per integer. In typical use, the
+ * consumption per integer is somewhere between those extremes, depending
+ * on the range of integers stored, and how "clustered" they are.
+ *
+ *
+ * Interface
+ * ---------
+ *
+ * intset_create - Create a new empty set.
+ * intset_add_member - Add an integer to the set.
+ * intset_is_member - Test if an integer is in the set
+ * intset_begin_iterate - Begin iterating through all integers in set
+ * intset_iterate_next - Return next integer
+ *
+ *
+ * Limitations
+ * -----------
+ *
+ * - Values must be added in order. (Random insertions would require
+ * splitting nodes, which hasn't been implemented.)
+ *
+ * - Values cannot be added while iteration is in progress.
+ *
+ * - No support for removing values.
+ *
+ * None of these limitations are fundamental to the data structure, and
+ * could be lifted if needed, by writing some new code. But the current
+ * users of this facility don't need them.
+ *
+ *
+ * References
+ * ----------
+ *
+ * Simple-8b encoding is based on:
+ *
+ * Vo Ngoc Anh , Alistair Moffat, Index compression using 64-bit words,
+ * Software - Practice & Experience, v.40 n.2, p.131-147, February 2010
+ * (https://doi.org/10.1002/spe.948)
+ *
+ *
+ * Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/lib/integerset.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "lib/integerset.h"
+#include "port/pg_bitutils.h"
+#include "utils/memutils.h"
+
+
+/*
+ * Properties of Simple-8b encoding. (These are needed here, before
+ * other definitions, so that we can size other arrays accordingly).
+ *
+ * SIMPLE8B_MAX_VALUE is the greatest integer that can be encoded. Simple-8B
+ * uses 64-bit words, but uses four bits to indicate the "mode" of the
+ * codeword, leaving at most 60 bits for the actual integer.
+ *
+ * SIMPLE8B_MAX_VALUES_PER_CODEWORD is the maximum number of integers that
+ * can be encoded in a single codeword.
+ */
+#define SIMPLE8B_MAX_VALUE ((1L << 60) - 1)
+#define SIMPLE8B_MAX_VALUES_PER_CODEWORD 240
+
+/*
+ * Parameters for shape of the in-memory B-tree.
+ *
+ * These set the size of each internal and leaf node. They don't necessarily
+ * need to be the same, because the tree is just an in-memory structure.
+ * With the default 64, each node is about 1 kb.
+ *
+ * If you change these, you must recalculate MAX_TREE_LEVELS, too!
+ */
+#define MAX_INTERNAL_ITEMS 64
+#define MAX_LEAF_ITEMS 64
+
+/*
+ * Maximum height of the tree.
+ *
+ * MAX_TREE_ITEMS is calculated from the "fan-out" of the B-tree. The
+ * theoretical maximum number of items that we can store in a set is 2^64,
+ * so MAX_TREE_LEVELS should be set so that:
+ *
+ * MAX_LEAF_ITEMS * MAX_INTERNAL_ITEMS ^ (MAX_TREE_LEVELS - 1) >= 2^64.
+ *
+ * In practice, we'll need far fewer levels, because you will run out of
+ * memory long before reaching that number, but let's be conservative.
+ */
+#define MAX_TREE_LEVELS 11
+
+/*
+ * Node structures, for the in-memory B-tree.
+ *
+ * An internal node holds a number of downlink pointers to leaf nodes, or
+ * to internal nodes on lower level. For each downlink, the key value
+ * corresponding the lower level node is stored in a sorted array. The
+ * stored key values are low keys. In other words, if the downlink has value
+ * X, then all items stored on that child are >= X.
+ *
+ * Each leaf node holds a number of "items", with a varying number of
+ * integers packed into each item. Each item consists of two 64-bit words:
+ * The first word holds first integer stored in the item, in plain format.
+ * The second word contains between 0 and 240 more integers, packed using
+ * Simple-8b encoding. By storing the first integer in plain, unpacked,
+ * format, we can use binary search to quickly find an item that holds (or
+ * would hold) a particular integer. And by storing the rest in packed form,
+ * we still get pretty good memory density, if there are clusters of integers
+ * with similar values.
+ *
+ * Each leaf node also has a pointer to the next leaf node, so that the leaf
+ * nodes can be easily walked from beginning to end, when iterating.
+ */
+typedef struct intset_node intset_node;
+typedef struct intset_leaf_node intset_leaf_node;
+typedef struct intset_internal_node intset_internal_node;
+
+/* Common structure of both leaf and internal nodes. */
+struct intset_node
+{
+ uint16 level;
+ uint16 num_items;
+};
+
+/* Internal node */
+struct intset_internal_node
+{
+ /* common header, must match intset_node */
+ uint16 level; /* >= 1 on internal nodes */
+ uint16 num_items;
+
+ /*
+ * 'values' is an array of key values, and 'downlinks' are pointers
+ * to lower-level nodes, corresponding to the key values.
+ */
+ uint64 values[MAX_INTERNAL_ITEMS];
+ intset_node *downlinks[MAX_INTERNAL_ITEMS];
+};
+
+/* Leaf node */
+typedef struct
+{
+ uint64 first; /* first integer in this item */
+ uint64 codeword; /* simple8b encoded differences from 'first' */
+} leaf_item;
+
+#define MAX_VALUES_PER_LEAF_ITEM (1 + SIMPLE8B_MAX_VALUES_PER_CODEWORD)
+
+struct intset_leaf_node
+{
+ /* common header, must match intset_node */
+ uint16 level; /* 0 on leafs */
+ uint16 num_items;
+
+ intset_leaf_node *next; /* right sibling, if any */
+
+ leaf_item items[MAX_LEAF_ITEMS];
+};
+
+/*
+ * We buffer insertions in a simple array, before packing and inserting them
+ * into the B-tree. MAX_BUFFERED_VALUES sets the size of the buffer. The
+ * encoder assumes that it is large enough, that we can always fill a leaf
+ * item with buffered new items. In other words, MAX_BUFFERED_VALUES must be
+ * larger than MAX_VALUES_PER_LEAF_ITEM.
+ */
+#define MAX_BUFFERED_VALUES (MAX_VALUES_PER_LEAF_ITEM * 2)
+
+/*
+ * IntegerSet is the top-level object representing the set.
+ *
+ * The integers are stored in an in-memory B-tree structure, and an array
+ * for newly-added integers. IntegerSet also tracks information about memory
+ * usage, as well as the current position, when iterating the set with
+ * intset_begin_iterate / intset_iterate_next.
+ */
+struct IntegerSet
+{
+ /*
+ * 'context' is a dedicated memory context, used to hold the IntegerSet
+ * struct itself, as well as all the tree nodes.
+ *
+ * 'mem_used' tracks the amount of memory used. We don't do anything with
+ * it in integerset.c itself, but the callers can ask for it with
+ * intset_memory_usage().
+ */
+ MemoryContext context; /* memory context holding everything */
+ uint64 mem_used; /* amount of memory used */
+
+ uint64 num_entries; /* total # of values in the set */
+ uint64 highest_value; /* highest value stored in this set */
+
+ /*
+ * B-tree to hold the packed values.
+ *
+ * 'rightmost_nodes' hold pointers to the rightmost node on each level.
+ * rightmost_parent[0] is rightmost leaf, rightmost_parent[1] is its
+ * parent, and so forth, all the way up to the root. These are needed when
+ * adding new values. (Currently, we require that new values are added at
+ * the end.)
+ */
+ int num_levels; /* height of the tree */
+ intset_node *root; /* root node */
+ intset_node *rightmost_nodes[MAX_TREE_LEVELS];
+ intset_leaf_node *leftmost_leaf; /* leftmost leaf node */
+
+ /*
+ * Holding area for new items that haven't been inserted to the tree yet.
+ */
+ uint64 buffered_values[MAX_BUFFERED_VALUES];
+ int num_buffered_values;
+
+ /*
+ * Iterator support.
+ *
+ * 'iter_values' is an array of integers ready to be returned to the
+ * caller. 'item_node' and 'item_itemno' point to the leaf node, and
+ * item within the leaf node, to get the next batch of values from.
+ *
+ * Normally, 'iter_values' points 'iter_values_buf', which holds items
+ * decoded from a leaf item. But after we have scanned the whole B-tree,
+ * we iterate through all the unbuffered values, too, by pointing
+ * iter_values to 'buffered_values'.
+ */
+ uint64 *iter_values;
+ int iter_num_values; /* number of elements in 'iter_values' */
+ int iter_valueno; /* index into 'iter_values' */
+ intset_leaf_node *iter_node; /* current leaf node */
+ int iter_itemno; /* next item 'iter_node' to decode */
+
+ uint64 iter_values_buf[MAX_VALUES_PER_LEAF_ITEM];
+};
+
+/*
+ * prototypes for internal functions.
+ */
+static void intset_update_upper(IntegerSet *intset, int level,
+ intset_node *new_node, uint64 new_node_item);
+static void intset_flush_buffered_values(IntegerSet *intset);
+
+static int intset_binsrch_uint64(uint64 value, uint64 *arr, int arr_elems,
+ bool nextkey);
+static int intset_binsrch_leaf(uint64 value, leaf_item *arr, int arr_elems,
+ bool nextkey);
+
+static uint64 simple8b_encode(uint64 *ints, int *num_encoded, uint64 base);
+static int simple8b_decode(uint64 codeword, uint64 *decoded, uint64 base);
+static bool simple8b_contains(uint64 codeword, uint64 key, uint64 base);
+
+
+/*
+ * Create a new, initially empty, integer set.
+ */
+IntegerSet *
+intset_create(void)
+{
+ MemoryContext context;
+ IntegerSet *intset;
+
+ /*
+ * Create a new memory context to hold everything.
+ *
+ * We never free any nodes, so the generational allocator works well for
+ * us.
+ *
+ * Use a large block size, in the hopes that if we use a lot of memory,
+ * the libc allocator will give it back to the OS when we free it, rather
+ * than add it to a free-list. (On glibc, see M_MMAP_THRESHOLD. As of this
+ * writing, the effective threshold is somewhere between 128 kB and 4 MB.)
+ */
+ context = GenerationContextCreate(CurrentMemoryContext,
+ "integer set",
+ SLAB_LARGE_BLOCK_SIZE);
+
+ /* Allocate the IntegerSet object itself, in the context. */
+ intset = (IntegerSet *) MemoryContextAlloc(context, sizeof(IntegerSet));
+ intset->context = context;
+ intset->mem_used = GetMemoryChunkSpace(intset);
+
+ intset->num_entries = 0;
+ intset->highest_value = 0;
+
+ intset->num_levels = 0;
+ intset->root = NULL;
+ memset(intset->rightmost_nodes, 0, sizeof(intset->rightmost_nodes));
+ intset->leftmost_leaf = NULL;
+
+ intset->num_buffered_values = 0;
+
+ intset->iter_node = NULL;
+ intset->iter_itemno = 0;
+ intset->iter_valueno = 0;
+ intset->iter_num_values = 0;
+
+ return intset;
+}
+
+/*
+ * Allocate a new node.
+ */
+static intset_internal_node *
+intset_new_internal_node(IntegerSet *intset)
+{
+ intset_internal_node *n;
+
+ n = (intset_internal_node *) MemoryContextAlloc(intset->context,
+ sizeof(intset_internal_node));
+ intset->mem_used += GetMemoryChunkSpace(n);
+
+ n->level = 0; /* caller must set */
+ n->num_items = 0;
+
+ return n;
+}
+
+static intset_leaf_node *
+intset_new_leaf_node(IntegerSet *intset)
+{
+ intset_leaf_node *n;
+
+ n = (intset_leaf_node *) MemoryContextAlloc(intset->context,
+ sizeof(intset_leaf_node));
+ intset->mem_used += GetMemoryChunkSpace(n);
+
+ n->level = 0;
+ n->num_items = 0;
+ n->next = NULL;
+
+ return n;
+}
+
+/*
+ * Free the integer set
+ */
+void
+intset_free(IntegerSet *intset)
+{
+ /* everything is allocated in the memory context */
+ MemoryContextDelete(intset->context);
+}
+
+/*
+ * Return the number of entries in the integer set.
+ */
+uint64
+intset_num_entries(IntegerSet *intset)
+{
+ return intset->num_entries;
+}
+
+/*
+ * Return the amount of memory used by the integer set.
+ */
+uint64
+intset_memory_usage(IntegerSet *intset)
+{
+ return intset->mem_used;
+}
+
+/*
+ * Add a value to the set.
+ *
+ * Values must be added in order.
+ */
+void
+intset_add_member(IntegerSet *intset, uint64 x)
+{
+ if (intset->iter_node)
+ elog(ERROR, "cannot add new values to integer set when iteration is in progress");
+
+ if (x <= intset->highest_value && intset->num_entries > 0)
+ elog(ERROR, "cannot add value to integer set out of order");
+
+ if (intset->num_buffered_values >= MAX_BUFFERED_VALUES)
+ {
+ /* Time to flush our buffer */
+ intset_flush_buffered_values(intset);
+ Assert(intset->num_buffered_values < MAX_BUFFERED_VALUES);
+ }
+
+ /* Add it to the buffer of newly-added values */
+ intset->buffered_values[intset->num_buffered_values] = x;
+ intset->num_buffered_values++;
+ intset->num_entries++;
+ intset->highest_value = x;
+}
+
+/*
+ * Take a batch of buffered values, and pack them into the B-tree.
+ */
+static void
+intset_flush_buffered_values(IntegerSet *intset)
+{
+ uint64 *values = intset->buffered_values;
+ uint64 num_values = intset->num_buffered_values;
+ int num_packed = 0;
+ intset_leaf_node *leaf;
+
+ leaf = (intset_leaf_node *) intset->rightmost_nodes[0];
+
+ /*
+ * If the tree is completely empty, create the first leaf page, which
+ * is also the root.
+ */
+ if (leaf == NULL)
+ {
+ /*
+ * This is the very first item in the set.
+ *
+ * Allocate root node. It's also a leaf.
+ */
+ leaf = intset_new_leaf_node(intset);
+
+ intset->root = (intset_node *) leaf;
+ intset->leftmost_leaf = leaf;
+ intset->rightmost_nodes[0] = (intset_node *) leaf;
+ intset->num_levels = 1;
+ }
+
+ /*
+ * If there are less than MAX_VALUES_PER_LEAF_ITEM values in the
+ * buffer, stop. In most cases, we cannot encode that many values
+ * in a single value, but this way, the encoder doesn't have to
+ * worry about running out of input.
+ */
+ while (num_values - num_packed >= MAX_VALUES_PER_LEAF_ITEM)
+ {
+ leaf_item item;
+ int num_encoded;
+
+ /*
+ * Construct the next leaf item, packing as many buffered values
+ * as possible.
+ */
+ item.first = values[num_packed];
+ item.codeword = simple8b_encode(&values[num_packed + 1],
+ &num_encoded,
+ item.first);
+
+ /*
+ * Add the item to the node, allocating a new node if the old one
+ * is full.
+ */
+ if (leaf->num_items >= MAX_LEAF_ITEMS)
+ {
+ /* Allocate new leaf and link it to the tree */
+ intset_leaf_node *old_leaf = leaf;
+
+ leaf = intset_new_leaf_node(intset);
+ old_leaf->next = leaf;
+ intset->rightmost_nodes[0] = (intset_node *) leaf;
+ intset_update_upper(intset, 1, (intset_node *) leaf, item.first);
+ }
+ leaf->items[leaf->num_items++] = item;
+
+ num_packed += 1 + num_encoded;
+ }
+
+ /*
+ * Move any remaining buffered values to the beginning of the array.
+ */
+ if (num_packed < intset->num_buffered_values)
+ {
+ memmove(&intset->buffered_values[0],
+ &intset->buffered_values[num_packed],
+ (intset->num_buffered_values - num_packed) * sizeof(uint64));
+ }
+ intset->num_buffered_values -= num_packed;
+}
+
+/*
+ * Insert a downlink into parent node, after creating a new node.
+ *
+ * Recurses if the parent node is full, too.
+ */
+static void
+intset_update_upper(IntegerSet *intset, int level, intset_node *child,
+ uint64 child_key)
+{
+ intset_internal_node *parent;
+
+ Assert(level > 0);
+
+ /*
+ * Create a new root node, if necessary.
+ */
+ if (level >= intset->num_levels)
+ {
+ intset_node *oldroot = intset->root;
+ uint64 downlink_key;
+
+ /* MAX_TREE_LEVELS should be more than enough, this shouldn't happen */
+ if (intset->num_levels == MAX_TREE_LEVELS)
+ elog(ERROR, "could not expand integer set, maximum number of levels reached");
+ intset->num_levels++;
+
+ /*
+ * Get the first value on the old root page, to be used as the
+ * downlink.
+ */
+ if (intset->root->level == 0)
+ downlink_key = ((intset_leaf_node *) oldroot)->items[0].first;
+ else
+ downlink_key = ((intset_internal_node *) oldroot)->values[0];
+
+ parent = intset_new_internal_node(intset);
+ parent->level = level;
+ parent->values[0] = downlink_key;
+ parent->downlinks[0] = oldroot;
+ parent->num_items = 1;
+
+ intset->root = (intset_node *) parent;
+ intset->rightmost_nodes[level] = (intset_node *) parent;
+ }
+
+ /*
+ * Place the downlink on the parent page.
+ */
+ parent = (intset_internal_node *) intset->rightmost_nodes[level];
+
+ if (parent->num_items < MAX_INTERNAL_ITEMS)
+ {
+ parent->values[parent->num_items] = child_key;
+ parent->downlinks[parent->num_items] = child;
+ parent->num_items++;
+ }
+ else
+ {
+ /*
+ * Doesn't fit. Allocate new parent, with the downlink as the first
+ * item on it, and recursively insert the downlink to the new parent
+ * to the grandparent.
+ */
+ parent = intset_new_internal_node(intset);
+ parent->level = level;
+ parent->values[0] = child_key;
+ parent->downlinks[0] = child;
+ parent->num_items = 1;
+
+ intset->rightmost_nodes[level] = (intset_node *) parent;
+
+ intset_update_upper(intset, level + 1, (intset_node *) parent, child_key);
+ }
+}
+
+/*
+ * Does the set contain the given value?
+ */
+bool
+intset_is_member(IntegerSet *intset, uint64 x)
+{
+ intset_node *node;
+ intset_leaf_node *leaf;
+ int level;
+ int itemno;
+ leaf_item *item;
+
+ /*
+ * The value might be in the buffer of newly-added values.
+ */
+ if (intset->num_buffered_values > 0 && x >= intset->buffered_values[0])
+ {
+ int itemno;
+
+ itemno = intset_binsrch_uint64(x,
+ intset->buffered_values,
+ intset->num_buffered_values,
+ false);
+ if (itemno >= intset->num_buffered_values)
+ return false;
+ else
+ return intset->buffered_values[itemno] == x;
+ }
+
+ /*
+ * Start from the root, and walk down the B-tree to find the right leaf
+ * node.
+ */
+ if (!intset->root)
+ return false;
+ node = intset->root;
+ for (level = intset->num_levels - 1; level > 0; level--)
+ {
+ intset_internal_node *n = (intset_internal_node *) node;
+
+ Assert(node->level == level);
+
+ itemno = intset_binsrch_uint64(x, n->values, n->num_items, true);
+ if (itemno == 0)
+ return false;
+ node = n->downlinks[itemno - 1];
+ }
+ Assert(node->level == 0);
+ leaf = (intset_leaf_node *) node;
+
+ /*
+ * Binary search the right item on the leaf page
+ */
+ itemno = intset_binsrch_leaf(x, leaf->items, leaf->num_items, true);
+ if (itemno == 0)
+ return false;
+ item = &leaf->items[itemno - 1];
+
+ /* Is this a match to the first value on the item? */
+ if (item->first == x)
+ return true;
+ Assert(x > item->first);
+
+ /* Is it in the packed codeword? */
+ if (simple8b_contains(item->codeword, x, item->first))
+ return true;
+
+ return false;
+}
+
+/*
+ * Begin in-order scan through all the values.
+ *
+ * While the iteration is in-progress, you cannot add new values to the set.
+ */
+void
+intset_begin_iterate(IntegerSet *intset)
+{
+ intset->iter_node = intset->leftmost_leaf;
+ intset->iter_itemno = 0;
+ intset->iter_valueno = 0;
+ intset->iter_num_values = 0;
+ intset->iter_values = intset->iter_values_buf;
+}
+
+/*
+ * Returns the next integer, when iterating.
+ *
+ * intset_begin_iterate() must be called first. intset_iterate_next() returns
+ * the next value in the set. If there are no more values, *found is set
+ * to false.
+ */
+uint64
+intset_iterate_next(IntegerSet *intset, bool *found)
+{
+ for (;;)
+ {
+ if (intset->iter_valueno < intset->iter_num_values)
+ {
+ *found = true;
+ return intset->iter_values[intset->iter_valueno++];
+ }
+
+ /* Our queue is empty, decode next leaf item */
+ if (intset->iter_node && intset->iter_itemno < intset->iter_node->num_items)
+ {
+ /* We have reached end of this packed item. Step to the next one. */
+ leaf_item *item;
+ int num_decoded;
+
+ item = &intset->iter_node->items[intset->iter_itemno++];
+
+ intset->iter_values[0] = item->first;
+ num_decoded = simple8b_decode(item->codeword, &intset->iter_values[1], item->first);
+ intset->iter_num_values = num_decoded + 1;
+
+ intset->iter_valueno = 0;
+ continue;
+ }
+
+ /* No more items on this leaf, step to next node */
+ if (intset->iter_node)
+ {
+ /* No more matches on this bucket. Step to the next node. */
+ intset->iter_node = intset->iter_node->next;
+ intset->iter_itemno = 0;
+ intset->iter_valueno = 0;
+ intset->iter_num_values = 0;
+ continue;
+ }
+
+ /*
+ * We have reached the end of the B-tree. But we might still have
+ * some integers in the buffer of newly-added values.
+ */
+ if (intset->iter_values == intset->iter_values_buf)
+ {
+ intset->iter_values = intset->buffered_values;
+ intset->iter_num_values = intset->num_buffered_values;
+ continue;
+ }
+
+ break;
+ }
+
+ /* No more results. */
+ *found = false;
+ return 0;
+}
+
+/*
+ * intset_binsrch_uint64() -- search a sorted array of uint64s
+ *
+ * Returns the first position with key equal or less than the given key.
+ * The returned position would be the "insert" location for the given key,
+ * that is, the position where the new key should be inserted to.
+ *
+ * 'nextkey' affects the behavior on equal keys. If true, and there is an
+ * equal key in the array, this returns the position immediately after the
+ * equal key. If false, this returns the position of the equal key itself.
+ */
+static int
+intset_binsrch_uint64(uint64 item, uint64 *arr, int arr_elems, bool nextkey)
+{
+ int low,
+ high,
+ mid;
+
+ low = 0;
+ high = arr_elems;
+ while (high > low)
+ {
+ mid = low + (high - low) / 2;
+
+ if (nextkey)
+ {
+ if (item >= arr[mid])
+ low = mid + 1;
+ else
+ high = mid;
+ }
+ else
+ {
+ if (item > arr[mid])
+ low = mid + 1;
+ else
+ high = mid;
+ }
+ }
+
+ return low;
+}
+
+/* same, but for an array of leaf items */
+static int
+intset_binsrch_leaf(uint64 item, leaf_item *arr, int arr_elems, bool nextkey)
+{
+ int low,
+ high,
+ mid;
+
+ low = 0;
+ high = arr_elems;
+ while (high > low)
+ {
+ mid = low + (high - low) / 2;
+
+ if (nextkey)
+ {
+ if (item >= arr[mid].first)
+ low = mid + 1;
+ else
+ high = mid;
+ }
+ else
+ {
+ if (item > arr[mid].first)
+ low = mid + 1;
+ else
+ high = mid;
+ }
+ }
+
+ return low;
+}
+
+/*
+ * Simple-8b encoding.
+ *
+ * Simple-8b algorithm packs between 1 and 240 integers into 64-bit words,
+ * called "codewords". The number of integers packed into a single codeword
+ * depends on the integers being packed: small integers are encoded using
+ * fewer bits than large integers. A single codeword can store a single
+ * 60-bit integer, or two 30-bit integers, for example.
+ *
+ * Since we're storing a unique, sorted, set of integers, we actually encode
+ * the *differences* between consecutive integers. That way, clusters of
+ * integers that are close to each other are packed efficiently, regardless
+ * of the absolute values.
+ *
+ * In Simple-8b, each codeword consists of a 4-bit selector, which indicates
+ * how many integers are encoded in the codeword, and the encoded integers
+ * packed into the remaining 60 bits. The selector allows for 16 different
+ * ways of using the remaining 60 bits, "modes". The number of integers
+ * packed into a single codeword is listed in the simple8b_modes table below.
+ * For example, consider the following codeword:
+ *
+ * 20-bit integer 20-bit integer 20-bit integer
+ * 1101 00000000000000010010 01111010000100100000 00000000000000010100
+ * ^
+ * selector
+ *
+ * The selector 1101 is 13 in decimal. From the modes table below, we see
+ * that it means that the codeword encodes three 12-bit integers. In decimal,
+ * those integers are 18, 500000 and 20. Because we encode deltas rather than
+ * absolute values, the actual values that they represent are 18, 500018 and
+ * 500038.
+ *
+ * Modes 0 and 1 are a bit special; they encode a run of 240 or 120 zeros
+ * (which means 240 or 120 consecutive integers, since we're encoding the
+ * the deltas between integers), without using the rest of the codeword bits
+ * for anything.
+ *
+ * Simple-8b cannot encode integers larger than 60 bits. Values larger than
+ * that are always stored in the 'first' field of a leaf item, never in the
+ * packed codeword. If there is a sequence of integers that are more than
+ * 2^60 apart, the codeword will go unused on those items. To represent that,
+ * we use a magic EMPTY_CODEWORD codeword.
+ */
+static const struct
+{
+ uint8 bits_per_int;
+ uint8 num_ints;
+} simple8b_modes[17] =
+{
+ { 0, 240 }, /* mode 0: 240 zeros */
+ { 0, 120 }, /* mode 1: 120 zeros */
+ { 1, 60 }, /* mode 2: sixty 1-bit integers */
+ { 2, 30 }, /* mode 3: thirty 2-bit integers */
+ { 3, 20 }, /* mode 4: twenty 3-bit integers */
+ { 4, 15 }, /* mode 5: fifteen 4-bit integers */
+ { 5, 12 }, /* mode 6: twelve 5-bit integers */
+ { 6, 10 }, /* mode 7: ten 6-bit integers */
+ { 7, 8 }, /* mode 8: eight 7-bit integers (four bits are wasted) */
+ { 8, 7 }, /* mode 9: seven 8-bit integers (four bits are wasted) */
+ { 10, 6 }, /* mode 10: six 10-bit integers */
+ { 12, 5 }, /* mode 11: five 12-bit integers */
+ { 15, 4 }, /* mode 12: four 15-bit integers */
+ { 20, 3 }, /* mode 13: three 20-bit integers */
+ { 30, 2 }, /* mode 14: two 30-bit integers */
+ { 60, 1 }, /* mode 15: one 60-bit integer */
+ { 0, 0 } /* sentinel value */
+};
+
+/*
+ * EMPTY_CODEWORD is a special value, used to indicate "no values".
+ * It is used if the next value is too large to be encoded with Simple-8b.
+ *
+ * This value looks like a 0-mode codeword, but we check for it
+ * specifically. (In a real 0-mode codeword, all the unused bits are zero.)
+ */
+#define EMPTY_CODEWORD (0xFFFFFFFFFFFFFFF0)
+
+/*
+ * Encode a number of integers into a Simple-8b codeword.
+ *
+ * Returns the number of integers that were encoded.
+ */
+static uint64
+simple8b_encode(uint64 *ints, int *num_encoded, uint64 base)
+{
+ int selector;
+ int nints;
+ int bits;
+ uint64 diff;
+ uint64 last_val;
+ uint64 codeword;
+ uint64 diffs[60];
+ int i;
+
+ Assert(ints[0] > base);
+
+ /*
+ * Select the "mode" to use for the next codeword.
+ *
+ * In each iteration, check if the next value can be represented
+ * in the current mode we're considering. If it's too large, then
+ * step up the mode to a wider one, and repeat. If it fits, move
+ * on to next integer. Repeat until the codeword is full, given
+ * the current mode.
+ *
+ * Note that we don't have any way to represent unused slots in the
+ * codeword, so we require each codeword to be "full".
+ */
+ selector = 0;
+ nints = simple8b_modes[0].num_ints;
+ bits = simple8b_modes[0].bits_per_int;
+ diff = ints[0] - base - 1;
+ last_val = ints[0];
+ i = 0;
+ for (;;)
+ {
+ if (diff >= (1L << bits))
+ {
+ /* too large, step up to next mode */
+ selector++;
+ nints = simple8b_modes[selector].num_ints;
+ bits = simple8b_modes[selector].bits_per_int;
+ if (i >= nints)
+ break;
+ }
+ else
+ {
+ if (i < 60)
+ diffs[i] = diff;
+ i++;
+ if (i >= nints)
+ break;
+
+ Assert(ints[i] > last_val);
+ diff = ints[i] - last_val - 1;
+ last_val = ints[i];
+ }
+ }
+
+ if (nints == 0)
+ {
+ /* The next value is too large and be encoded with Simple-8b */
+ Assert(i == 0);
+ *num_encoded = 0;
+ return EMPTY_CODEWORD;
+ }
+
+ /*
+ * Encode the integers using the selected mode. Note that we shift them
+ * into the codeword in reverse order, so that they will come out in the
+ * correct order in the decoder.
+ */
+ codeword = 0;
+ if (bits > 0)
+ {
+ for (i = nints - 1; i >= 0; i--)
+ {
+ codeword <<= bits;
+ codeword |= diffs[i];
+ }
+ }
+
+ /* add selector to the codeword, and return */
+ codeword <<= 4;
+ codeword |= selector;
+
+ *num_encoded = nints;
+ return codeword;
+}
+
+/*
+ * Decode a codeword into an array of integers.
+ */
+static int
+simple8b_decode(uint64 codeword, uint64 *decoded, uint64 base)
+{
+ int selector = codeword & 0x0f;
+ int nints = simple8b_modes[selector].num_ints;
+ uint64 bits = simple8b_modes[selector].bits_per_int;
+ uint64 mask = (1L << bits) - 1;
+ uint64 prev_value;
+
+ if (codeword == EMPTY_CODEWORD)
+ return 0;
+
+ codeword >>= 4; /* shift out the selector */
+
+ prev_value = base;
+ for (int i = 0; i < nints; i++)
+ {
+ uint64 diff = codeword & mask;
+
+ decoded[i] = prev_value + 1L + diff;
+ prev_value = decoded[i];
+ codeword >>= bits;
+ }
+ return nints;
+}
+
+/*
+ * This is very similar to simple8b_decode(), but instead of decoding all
+ * the values to an array, it just checks if the given integer is part of
+ * the codeword.
+ */
+static bool
+simple8b_contains(uint64 codeword, uint64 key, uint64 base)
+{
+ int selector = codeword & 0x0f;
+ int nints = simple8b_modes[selector].num_ints;
+ int bits = simple8b_modes[selector].bits_per_int;
+
+ if (codeword == EMPTY_CODEWORD)
+ return false;
+
+ codeword >>= 4; /* shift out the selector */
+
+ if (bits == 0)
+ {
+ /* Special handling for 0-bit cases. */
+ return key - base <= nints;
+ }
+ else
+ {
+ int mask = (1L << bits) - 1;
+ uint64 prev_value;
+
+ prev_value = base;
+ for (int i = 0; i < nints; i++)
+ {
+ uint64 diff = codeword & mask;
+ uint64 curr_value;
+
+ curr_value = prev_value + 1L + diff;
+
+ if (curr_value >= key)
+ {
+ if (curr_value == key)
+ return true;
+ else
+ return false;
+ }
+
+ codeword >>= bits;
+ prev_value = curr_value;
+ }
+ }
+ return false;
+}
diff --git a/src/include/lib/integerset.h b/src/include/lib/integerset.h
new file mode 100644
index 00000000000..27aa3ee883c
--- /dev/null
+++ b/src/include/lib/integerset.h
@@ -0,0 +1,25 @@
+/*
+ * integerset.h
+ * In-memory data structure to hold a large set of integers efficiently
+ *
+ * Portions Copyright (c) 2012-2019, PostgreSQL Global Development Group
+ *
+ * src/include/lib/integerset.h
+ */
+#ifndef INTEGERSET_H
+#define INTEGERSET_H
+
+typedef struct IntegerSet IntegerSet;
+
+extern IntegerSet *intset_create(void);
+extern void intset_free(IntegerSet *intset);
+extern void intset_add_member(IntegerSet *intset, uint64 x);
+extern bool intset_is_member(IntegerSet *intset, uint64 x);
+
+extern uint64 intset_num_entries(IntegerSet *intset);
+extern uint64 intset_memory_usage(IntegerSet *intset);
+
+extern void intset_begin_iterate(IntegerSet *intset);
+extern uint64 intset_iterate_next(IntegerSet *intset, bool *found);
+
+#endif /* INTEGERSET_H */
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 19d60a506e1..dfd0956aee3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -12,6 +12,7 @@ SUBDIRS = \
test_bloomfilter \
test_ddl_deparse \
test_extensions \
+ test_integerset \
test_parser \
test_pg_dump \
test_predtest \
diff --git a/src/test/modules/test_integerset/.gitignore b/src/test/modules/test_integerset/.gitignore
new file mode 100644
index 00000000000..5dcb3ff9723
--- /dev/null
+++ b/src/test/modules/test_integerset/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_integerset/Makefile b/src/test/modules/test_integerset/Makefile
new file mode 100644
index 00000000000..3b7c4999d6f
--- /dev/null
+++ b/src/test/modules/test_integerset/Makefile
@@ -0,0 +1,21 @@
+# src/test/modules/test_integerset/Makefile
+
+MODULE_big = test_integerset
+OBJS = test_integerset.o $(WIN32RES)
+PGFILEDESC = "test_integerset - test code for src/backend/lib/integerset.c"
+
+EXTENSION = test_integerset
+DATA = test_integerset--1.0.sql
+
+REGRESS = test_integerset
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_integerset
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_integerset/README b/src/test/modules/test_integerset/README
new file mode 100644
index 00000000000..3e4226adb55
--- /dev/null
+++ b/src/test/modules/test_integerset/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation,
+in src/backend/lib/integerset.c
+
+The tests verify the correctness of the implemention, but they can also be
+as a micro-benchmark: If you set the 'intset_tests_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_integerset/expected/test_integerset.out b/src/test/modules/test_integerset/expected/test_integerset.out
new file mode 100644
index 00000000000..d7c88ded092
--- /dev/null
+++ b/src/test/modules/test_integerset/expected/test_integerset.out
@@ -0,0 +1,14 @@
+CREATE EXTENSION test_integerset;
+--
+-- These tests don't produce any interesting output. We're checking that
+-- the operations complete without crashing or hanging and that none of their
+-- internal sanity tests fail. They print progress information as INFOs,
+-- which are not interesting for automated tests, so suppress those.
+--
+SET client_min_messages = 'warning';
+SELECT test_integerset();
+ test_integerset
+-----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_integerset/sql/test_integerset.sql b/src/test/modules/test_integerset/sql/test_integerset.sql
new file mode 100644
index 00000000000..34223afa885
--- /dev/null
+++ b/src/test/modules/test_integerset/sql/test_integerset.sql
@@ -0,0 +1,11 @@
+CREATE EXTENSION test_integerset;
+
+--
+-- These tests don't produce any interesting output. We're checking that
+-- the operations complete without crashing or hanging and that none of their
+-- internal sanity tests fail. They print progress information as INFOs,
+-- which are not interesting for automated tests, so suppress those.
+--
+SET client_min_messages = 'warning';
+
+SELECT test_integerset();
diff --git a/src/test/modules/test_integerset/test_integerset--1.0.sql b/src/test/modules/test_integerset/test_integerset--1.0.sql
new file mode 100644
index 00000000000..d6d5a3f6cf7
--- /dev/null
+++ b/src/test/modules/test_integerset/test_integerset--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_integerset/test_integerset--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_integerset" to load this file. \quit
+
+CREATE FUNCTION test_integerset()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_integerset/test_integerset.c b/src/test/modules/test_integerset/test_integerset.c
new file mode 100644
index 00000000000..24a2e08c0d1
--- /dev/null
+++ b/src/test/modules/test_integerset/test_integerset.c
@@ -0,0 +1,622 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_integerset.c
+ * Test integer set data structure.
+ *
+ * Copyright (c) 2019, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_integerset/test_integerset.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "fmgr.h"
+#include "lib/integerset.h"
+#include "nodes/bitmapset.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "miscadmin.h"
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns.
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool intset_test_stats = true;
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_integerset);
+
+/*
+ * A struct to define a pattern of integers, for use with test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 100000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 100000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 10000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 100000000
+ },
+ {
+ "clusters of thousand",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 100000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 10000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 10000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ 10000000000L, 1000000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ 10000000000L, 10000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ 2000000000000000000L, 23 /* can't be much higher than this, or we overflow uint64 */
+ }
+};
+
+static void test_pattern(const test_spec *spec);
+static void test_empty(void);
+static void test_single_value(uint64 value);
+static void check_with_filler(IntegerSet *intset, uint64 x, uint64 value, uint64 filler_min, uint64 filler_max);
+static void test_single_value_and_filler(uint64 value, uint64 filler_min, uint64 filler_max);
+static void test_huge_distances(void);
+
+/*
+ * SQL-callable entry point to perform all tests.
+ */
+Datum
+test_integerset(PG_FUNCTION_ARGS)
+{
+ test_huge_distances();
+
+ test_empty();
+
+ test_single_value(0);
+ test_single_value(1);
+ test_single_value(PG_UINT64_MAX - 1);
+ test_single_value(PG_UINT64_MAX);
+
+ /* Same tests, but with some "filler" values, so that the B-tree gets created */
+ test_single_value_and_filler(0, 1000, 2000);
+ test_single_value_and_filler(1, 1000, 2000);
+ test_single_value_and_filler(1, 1000, 2000000);
+ test_single_value_and_filler(PG_UINT64_MAX - 1, 1000, 2000);
+ test_single_value_and_filler(PG_UINT64_MAX, 1000, 2000);
+
+ test_huge_distances();
+
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec *spec)
+{
+ IntegerSet *intset;
+ MemoryContext test_cxt;
+ MemoryContext old_cxt;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+
+ elog(NOTICE, "testing intset with pattern \"%s\"", spec->test_name);
+ if (intset_test_stats)
+ fprintf(stderr, "-----\ntesting intset with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the integer set.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily. (intset_create() creates a memory context of its
+ * own, too, but we don't have direct access to it, so we cannot call
+ * MemoryContextStats() on it directly).
+ */
+ test_cxt = AllocSetContextCreate(CurrentMemoryContext,
+ "intset test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(test_cxt, spec->test_name);
+ old_cxt = MemoryContextSwitchTo(test_cxt);
+ intset = intset_create();
+ MemoryContextSwitchTo(old_cxt);
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ x = last_int + pattern_values[i];
+
+ intset_add_member(intset, x);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (intset_test_stats)
+ fprintf(stderr, "added %lu values in %lu ms\n",
+ spec->num_values, (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by intset_memory_usage(), as well as the
+ * stats from the memory context. They should be in the same ballpark,
+ * but it's hard to automate testing that, so if you're making changes
+ * to the implementation, just observe that manually.
+ */
+ if (intset_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by intset_memory_usage().
+ * It should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = intset_memory_usage(intset);
+ fprintf(stderr, "intset_memory_usage() reported %lu (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(test_cxt);
+ }
+
+ /* Check that intset_get_num_entries works */
+ n = intset_num_entries(intset);
+ if (n != spec->num_values)
+ elog(ERROR, "intset_num_entries returned %lu, expected %lu", n, spec->num_values);
+
+ /*
+ * Test random-access probes with intset_is_member()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 1000000; n++)
+ {
+ bool b;
+ bool expected;
+ uint64 x;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit hit values that are actually in the set.
+ */
+ x = (pg_lrand48() << 31) | pg_lrand48();
+ x = x % (last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to intset_is_member() ? */
+ b = intset_is_member(intset, x);
+
+ if (b != expected)
+ elog(ERROR, "mismatch at %lu: %d vs %d", x, b, expected);
+ }
+ endtime = GetCurrentTimestamp();
+ if (intset_test_stats)
+ fprintf(stderr, "probed %lu values in %lu ms\n", n, (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ intset_begin_iterate(intset);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ bool found;
+
+ x = intset_iterate_next(intset, &found);
+ if (!found)
+ break;
+
+ if (x != expected)
+ elog(ERROR, "iterate returned wrong value; got %lu, expected %lu", x, expected);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (intset_test_stats)
+ fprintf(stderr, "iterated %lu values in %lu ms\n", n, (endtime - starttime) / 1000);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after %lu entries, expected %lu", n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned %lu entries, %lu was expected", n, spec->num_values);
+
+ intset_free(intset);
+}
+
+/*
+ * Test with a set containing a single integer.
+ */
+static void
+test_single_value(uint64 value)
+{
+ IntegerSet *intset;
+ uint64 x;
+ uint64 num_entries;
+ bool found;
+
+ elog(NOTICE, "testing intset with single value %lu", value);
+
+ /* Create the set. */
+ intset = intset_create();
+ intset_add_member(intset, value);
+
+ /* Test intset_get_num_entries() */
+ num_entries = intset_num_entries(intset);
+ if (num_entries != 1)
+ elog(ERROR, "intset_num_entries returned %lu, expected %lu", num_entries, 1L);
+
+ /*
+ * Test intset_is_member() at various special values, like 0 and and maximum
+ * possible 64-bit integer, as well as the value itself.
+ */
+ if (intset_is_member(intset, 0) != (value == 0))
+ elog(ERROR, "intset_is_member failed for 0");
+ if (intset_is_member(intset, 1) != (value == 1))
+ elog(ERROR, "intset_is_member failed for 1");
+ if (intset_is_member(intset, PG_UINT64_MAX) != (value == PG_UINT64_MAX))
+ elog(ERROR, "intset_is_member failed for PG_UINT64_MAX");
+ if (intset_is_member(intset, value) != true)
+ elog(ERROR, "intset_is_member failed for the tested value");
+
+ /*
+ * Test iterator
+ */
+ intset_begin_iterate(intset);
+ x = intset_iterate_next(intset, &found);
+ if (!found || x != value)
+ elog(ERROR, "intset_iterate_next failed for %lu", x);
+
+ x = intset_iterate_next(intset, &found);
+ if (found)
+ elog(ERROR, "intset_iterate_next failed %lu", x);
+
+ intset_free(intset);
+}
+
+/*
+ * Test with an integer set that contains:
+ *
+ * - a given single 'value', and
+ * - all integers between 'filler_min' and 'filler_max'.
+ *
+ * This exercises different codepaths than testing just with a single value,
+ * because the implementation buffers newly-added values. If we add just
+ * single value to the set, we won't test the internal B-tree code at all,
+ * just the code that deals with the buffer.
+ */
+static void
+test_single_value_and_filler(uint64 value, uint64 filler_min, uint64 filler_max)
+{
+ IntegerSet *intset;
+ uint64 x;
+ bool found;
+ uint64 *iter_expected;
+ uint64 n = 0;
+ uint64 num_entries = 0;
+ uint64 mem_usage;
+
+ elog(NOTICE, "testing intset with value %lu, and all between %lu and %lu",
+ value, filler_min, filler_max);
+
+ intset = intset_create();
+
+ iter_expected = palloc(sizeof(uint64) * (filler_max - filler_min + 1));
+ if (value < filler_min)
+ {
+ intset_add_member(intset, value);
+ iter_expected[n++] = value;
+ }
+
+ for (x = filler_min; x < filler_max; x++)
+ {
+ intset_add_member(intset, x);
+ iter_expected[n++] = x;
+ }
+
+ if (value >= filler_max)
+ {
+ intset_add_member(intset, value);
+ iter_expected[n++] = value;
+ }
+
+ /* Test intset_get_num_entries() */
+ num_entries = intset_num_entries(intset);
+ if (num_entries != n)
+ elog(ERROR, "intset_num_entries returned %lu, expected %lu", num_entries, n);
+
+ /*
+ * Test intset_is_member() at various spots, at and around the values that we
+ * expect to be set, as well as 0 and the maximum possible value.
+ */
+ check_with_filler(intset, 0, value, filler_min, filler_max);
+ check_with_filler(intset, 1, value, filler_min, filler_max);
+ check_with_filler(intset, filler_min - 1, value, filler_min, filler_max);
+ check_with_filler(intset, filler_min, value, filler_min, filler_max);
+ check_with_filler(intset, filler_min + 1, value, filler_min, filler_max);
+ check_with_filler(intset, value - 1, value, filler_min, filler_max);
+ check_with_filler(intset, value, value, filler_min, filler_max);
+ check_with_filler(intset, value + 1, value, filler_min, filler_max);
+ check_with_filler(intset, filler_max - 1, value, filler_min, filler_max);
+ check_with_filler(intset, filler_max, value, filler_min, filler_max);
+ check_with_filler(intset, filler_max + 1, value, filler_min, filler_max);
+ check_with_filler(intset, PG_UINT64_MAX - 1, value, filler_min, filler_max);
+ check_with_filler(intset, PG_UINT64_MAX, value, filler_min, filler_max);
+
+ intset_begin_iterate(intset);
+ for (uint64 i = 0; i < n; i++)
+ {
+ x = intset_iterate_next(intset, &found);
+ if (!found || x != iter_expected[i])
+ elog(ERROR, "intset_iterate_next failed for %lu", x);
+ }
+ x = intset_iterate_next(intset, &found);
+ if (found)
+ elog(ERROR, "intset_iterate_next failed %lu", x);
+
+ mem_usage = intset_memory_usage(intset);
+ if (mem_usage < 5000 || mem_usage > 500000000)
+ elog(ERROR, "intset_memory_usage() reported suspicous value: %lu", mem_usage);
+
+ intset_free(intset);
+}
+
+/*
+ * Helper function for test_single_value_and_filler.
+ *
+ * Calls intset_is_member() for value 'x', and checks that the result is what
+ * we expect.
+ */
+static void
+check_with_filler(IntegerSet *intset, uint64 x,
+ uint64 value, uint64 filler_min, uint64 filler_max)
+{
+ bool expected;
+ bool actual;
+
+ expected = (x == value || (filler_min <= x && x < filler_max));
+
+ actual = intset_is_member(intset, x);
+
+ if (actual != expected)
+ elog(ERROR, "intset_is_member failed for %lu", x);
+}
+
+/*
+ * Test empty set
+ */
+static void
+test_empty(void)
+{
+ IntegerSet *intset;
+ bool found = true;
+ uint64 x;
+
+ elog(NOTICE, "testing intset with empty set");
+
+ intset = intset_create();
+
+ /* Test intset_is_member() */
+ if (intset_is_member(intset, 0) != false)
+ elog(ERROR, "intset_is_member on empty set returned true");
+ if (intset_is_member(intset, 1) != false)
+ elog(ERROR, "intset_is_member on empty set returned true");
+ if (intset_is_member(intset, PG_UINT64_MAX) != false)
+ elog(ERROR, "intset_is_member on empty set returned true");
+
+ /* Test iterator */
+ intset_begin_iterate(intset);
+ x = intset_iterate_next(intset, &found);
+ if (found)
+ elog(ERROR, "intset_iterate_next on empty set returned a value (%lu)", x);
+
+ intset_free(intset);
+}
+
+/*
+ * Test with integers that are more than 2^60 apart.
+ *
+ * The Simple-8b encoding used by the set implementation can only encode
+ * values up to 2^60. That makes large differences like this interesting
+ * to test.
+ */
+static void
+test_huge_distances(void)
+{
+ IntegerSet *intset;
+ uint64 values[1000];
+ int num_values = 0;
+ uint64 val = 0;
+ bool found;
+ uint64 x;
+
+ elog(NOTICE, "testing intset with distances > 2^60 between values");
+
+ val = 0;
+ values[num_values++] = val;
+
+ val += 1152921504606846976L - 1; /* 2^60 - 1*/
+ values[num_values++] = val;
+
+ val += 1152921504606846976L - 1; /* 2^60 - 1*/
+ values[num_values++] = val;
+
+ val += 1152921504606846976L; /* 2^60 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L; /* 2^60 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L; /* 2^60 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L + 1; /* 2^60 + 1 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L + 1; /* 2^60 + 1 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L + 1; /* 2^60 + 1 */
+ values[num_values++] = val;
+
+ val += 1152921504606846976L; /* 2^60 */
+ values[num_values++] = val;
+
+ /* we're now very close to 2^64, so can't add large values anymore */
+
+ intset = intset_create();
+
+ /*
+ * Add many more values to the end, to make sure that all the above
+ * values get flushed and packed into the tree structure.
+ */
+ while (num_values < 1000)
+ {
+ val += pg_lrand48();
+ values[num_values++] = val;
+ }
+
+ /* Add these numbers to the set */
+ for (int i = 0; i < num_values; i++)
+ intset_add_member(intset, values[i]);
+
+ /*
+ * Test iterset_is_member() around each of these values
+ */
+ for (int i = 0; i < num_values; i++)
+ {
+ uint64 x = values[i];
+ bool result;
+
+ if (x > 0)
+ {
+ result = intset_is_member(intset, x - 1);
+ if (result != false)
+ elog(ERROR, "intset_is_member failed for %lu", x - 1);
+ }
+
+ result = intset_is_member(intset, x);
+ if (result != true)
+ elog(ERROR, "intset_is_member failed for %lu", x);
+
+ result = intset_is_member(intset, x + 1);
+ if (result != false)
+ elog(ERROR, "intset_is_member failed for %lu", x + 1);
+ }
+
+ /*
+ * Test iterator
+ */
+ intset_begin_iterate(intset);
+ for (int i = 0; i < num_values; i++)
+ {
+ x = intset_iterate_next(intset, &found);
+ if (!found || x != values[i])
+ elog(ERROR, "intset_iterate_next failed for %lu", x);
+ }
+ x = intset_iterate_next(intset, &found);
+ if (found)
+ elog(ERROR, "intset_iterate_next failed %lu", x);
+}
diff --git a/src/test/modules/test_integerset/test_integerset.control b/src/test/modules/test_integerset/test_integerset.control
new file mode 100644
index 00000000000..7d20c2d7b88
--- /dev/null
+++ b/src/test/modules/test_integerset/test_integerset.control
@@ -0,0 +1,4 @@
+comment = 'Test code for integerset'
+default_version = '1.0'
+module_pathname = '$libdir/test_integerset'
+relocatable = true
--
2.20.1
--------------68960491603935AE18105C47
Content-Type: text/x-patch;
name="0002-Delete-empty-pages-during-GiST-VACUUM.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="0002-Delete-empty-pages-during-GiST-VACUUM.patch"
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v4 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 51 ++++++++++++++++++-------
doc/src/sgml/archive-modules.sgml | 33 +++++++++++++---
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 112 insertions(+), 47 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..3b4373b562 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +86,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void
+basic_archive_startup(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = palloc0(sizeof(BasicArchiveData));
+
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -135,7 +158,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +169,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2ec5d12d97 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
+ <function>_PG_archive_module_init</function>. This function must return an
+ ArchiveModuleCallbacks struct filled with the callback function pointers for
individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <literal>state->private_data</literal> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ec47b2cc20..5931d5ffa0 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 35f88c1222..cc2585218d 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,28 +23,33 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -126,7 +131,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--9amGYk9869ThD9tj--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v5 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 53 +++++++++++++++++++------
doc/src/sgml/archive-modules.sgml | 33 ++++++++++++---
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 114 insertions(+), 47 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..74a51b9288 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +86,30 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void
+basic_archive_startup(ArchiveModuleState *state)
+{
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -135,7 +160,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +171,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2ec5d12d97 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
+ <function>_PG_archive_module_init</function>. This function must return an
+ ArchiveModuleCallbacks struct filled with the callback function pointers for
individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <literal>state->private_data</literal> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ec47b2cc20..5931d5ffa0 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 35f88c1222..cc2585218d 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,28 +23,33 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -126,7 +131,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--+QahgC5+KEYLbs62--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v2 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 42 +++++++++++++++++--------
doc/src/sgml/archive-modules.sgml | 30 +++++++++++++-----
src/backend/postmaster/pgarch.c | 22 +++++++------
src/backend/postmaster/shell_archive.c | 29 ++++++++++-------
src/include/postmaster/archive_module.h | 14 +++++----
5 files changed, 89 insertions(+), 48 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..2b7680947f 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -41,14 +41,21 @@
PG_MODULE_MAGIC;
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void *basic_archive_startup(void);
+static bool basic_archive_configured(void *private_state);
+static bool basic_archive_file(const char *file, const char *path, void *private_state);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +74,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +81,25 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void *
+basic_archive_startup(void)
+{
+ return (void *) AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -135,7 +150,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(void *private_state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +161,11 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(const char *file, const char *path, void *private_state)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ MemoryContext basic_archive_context = (MemoryContext) private_state;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..8c4a3d9365 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,18 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. This function must return a
+ struct filled with the callback function pointers for individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +73,22 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs a state, it should
+ return a pointer to the state. This pointer will be passed to each of the
+ module's other callbacks via the <literal>void *private_state</literal>
+ argument.
+
+<programlisting>
+typedef void *(*ArchiveStartupCB) (void);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +99,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (void *private_state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +121,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (const char *file, const char *path, void *private_state);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +144,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (void *private_state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index eca02d5e74..421425d0dd 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -98,7 +98,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static void *private_state = NULL;
/*
@@ -417,8 +418,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(private_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -519,7 +520,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(xlog, pathname, private_state);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -838,8 +839,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -855,12 +854,15 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ private_state = ArchiveCallbacks->startup_cb();
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -870,6 +872,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(private_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index b64297e3bb..dbd633477a 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -22,28 +22,33 @@
#include "pgstat.h"
#include "postmaster/archive_module.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(void *private_state);
+static bool shell_archive_file(const char *file, const char *path, void *private_state);
+static void shell_archive_shutdown(void *private_state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(void *private_state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(const char *file, const char *path, void *private_state)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -125,7 +130,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(void *private_state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index 099050c1ca..b4ea4f5dcd 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -25,12 +25,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void *(*ArchiveStartupCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (void *private_state);
+typedef bool (*ArchiveFileCB) (const char *file, const char *path, void *private_state);
+typedef void (*ArchiveShutdownCB) (void *private_state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,15 +42,15 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
/*
* Since the logic for archiving via a shell command is in the core server
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _ARCHIVE_MODULE_H */
--
2.25.1
--cWoXeonUoKmBZSoM--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v6 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 53 +++++++++++++++++++------
doc/src/sgml/archive-modules.sgml | 35 ++++++++++++----
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 115 insertions(+), 48 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..74a51b9288 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +86,30 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void
+basic_archive_startup(ArchiveModuleState *state)
+{
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -135,7 +160,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +171,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2db1b19216 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. This function must return an
+ <structname>ArchiveModuleCallbacks</structname> struct filled with the
+ callback function pointers for individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 281d9fd8b7..99a07996f7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 35f88c1222..cc2585218d 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,28 +23,33 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -126,7 +131,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--OXfL5xGRrasGEqWY--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v7 3/4] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 53 +++++++++++++++++++------
doc/src/sgml/archive-modules.sgml | 35 ++++++++++++----
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 115 insertions(+), 48 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..74a51b9288 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +86,30 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void
+basic_archive_startup(ArchiveModuleState *state)
+{
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -135,7 +160,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +171,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2db1b19216 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. This function must return an
+ <structname>ArchiveModuleCallbacks</structname> struct filled with the
+ callback function pointers for individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 281d9fd8b7..99a07996f7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 35f88c1222..cc2585218d 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,28 +23,33 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -126,7 +131,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--/04w6evG8XlLl3ft
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0004-remove-unnecessary-assertions-for-functions-with-.patch"
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v8 4/4] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 51 +++++++++++++++++++------
doc/src/sgml/archive-modules.sgml | 35 +++++++++++++----
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 114 insertions(+), 47 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 384336884d..9afa03c5e2 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +86,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +158,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +169,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2db1b19216 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. This function must return an
+ <structname>ArchiveModuleCallbacks</structname> struct filled with the
+ callback function pointers for individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 281d9fd8b7..99a07996f7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 0f0558e155..54b02ca152 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,26 +23,31 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -124,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--4Ckj6UjgE2iN1+kY--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v9 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 51 +++++++++++++++++++------
doc/src/sgml/archive-modules.sgml | 35 +++++++++++++----
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 114 insertions(+), 47 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 384336884d..9afa03c5e2 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +86,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +158,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +169,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..2db1b19216 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. This function must return an
+ <structname>ArchiveModuleCallbacks</structname> struct filled with the
+ callback function pointers for individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 281d9fd8b7..99a07996f7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 0f0558e155..54b02ca152 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,26 +23,31 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -124,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..79e9d53534 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index 3cf737ce8e..be83c7715a 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -19,6 +19,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--2oS5YaxWCcQjTEyO--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v3 3/3] restructure archive modules API
@ 2023-01-31 22:58 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-01-31 22:58 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 51 ++++++++++++++++++-------
doc/src/sgml/archive-modules.sgml | 33 +++++++++++++---
src/backend/postmaster/pgarch.c | 23 ++++++-----
src/backend/postmaster/shell_archive.c | 29 ++++++++------
src/include/postmaster/archive_module.h | 21 +++++++---
src/include/postmaster/shell_archive.h | 2 +-
6 files changed, 112 insertions(+), 47 deletions(-)
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 87bbb2174d..f36a1def15 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(const char *file, const char *path, ArchiveModuleState *state);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,13 +86,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
-void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
{
AssertVariableIsOfType(&_PG_archive_module_init, ArchiveModuleInit);
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
+void
+basic_archive_startup(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = palloc0(sizeof(BasicArchiveData));
+
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -135,7 +158,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -146,10 +169,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(const char *file, const char *path, ArchiveModuleState *state)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..cc8a834a2e 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,23 +47,30 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
+ <function>_PG_archive_module_init</function>. This function must return an
+ ArchiveModuleCallbacks struct filled with the callback function pointers for
individual actions.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
others are optional.
</para>
+
+ <note>
+ <para>
+ <varname>archive_library</varname> is only loaded in the archiver process.
+ </para>
+ </note>
</sect1>
<sect1 id="archive-module-callbacks">
@@ -73,6 +80,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module needs to have a state, it
+ can use <literal>state->private_data</literal> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +104,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +126,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (const char *file, const char *path, ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +149,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ec47b2cc20..253d456837 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -99,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveCallbacks;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -408,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveCallbacks.check_configured_cb != NULL &&
- !ArchiveCallbacks.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -510,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveCallbacks.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(xlog, pathname, archive_module_state);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -829,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveCallbacks, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -846,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveCallbacks);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveCallbacks.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -861,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveCallbacks.shutdown_cb != NULL)
- ArchiveCallbacks.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 35f88c1222..5eda7671e8 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -23,28 +23,33 @@
#include "postmaster/archive_module.h"
#include "postmaster/shell_archive.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(const char *file, const char *path, ArchiveModuleState *state);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
AssertVariableIsOfType(&shell_archive_init, ArchiveModuleInit);
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(const char *file, const char *path, ArchiveModuleState *state)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -126,7 +131,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
index f5015ff95d..d07dd26c6e 100644
--- a/src/include/postmaster/archive_module.h
+++ b/src/include/postmaster/archive_module.h
@@ -17,6 +17,15 @@
*/
extern PGDLLIMPORT char *XLogArchiveLibrary;
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
/*
* Archive module callbacks
*
@@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary;
* For more information about the purpose of each callback, refer to the
* archive modules documentation.
*/
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (const char *file, const char *path, ArchiveModuleState *state);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
@@ -40,8 +51,8 @@ typedef struct ArchiveModuleCallbacks
* Type of the shared library symbol _PG_archive_module_init that is looked
* up when loading an archive library.
*/
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
index f6943c0aba..66b2fda401 100644
--- a/src/include/postmaster/shell_archive.h
+++ b/src/include/postmaster/shell_archive.h
@@ -17,6 +17,6 @@
* and does not need to be loaded via a shared library, it has a special
* initialization function.
*/
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--sm4nu43k4a2Rpi4c--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v10 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 91 +++++++++++++++++++++----
doc/src/sgml/archive-modules.sgml | 35 +++++++---
src/backend/postmaster/pgarch.c | 27 +++++---
src/backend/postmaster/shell_archive.c | 34 +++++----
src/backend/utils/misc/guc_tables.c | 1 +
src/include/postmaster/archive_module.h | 58 ++++++++++++++++
src/include/postmaster/pgarch.h | 39 -----------
src/include/postmaster/shell_archive.h | 24 +++++++
8 files changed, 224 insertions(+), 85 deletions(-)
create mode 100644 src/include/postmaster/archive_module.h
create mode 100644 src/include/postmaster/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..e4742c9c94 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -32,7 +32,7 @@
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
+#include "postmaster/archive_module.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,14 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static void basic_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = basic_archive_shutdown
+};
/*
* _PG_init
@@ -67,10 +80,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +87,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +159,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +170,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
@@ -366,3 +394,40 @@ compare_files(const char *file1, const char *file2)
return ret;
}
+
+/*
+ * basic_archive_shutdown
+ *
+ * Frees our allocated state.
+ */
+static void
+basic_archive_shutdown(ArchiveModuleState *state)
+{
+ BasicArchiveData *data;
+ MemoryContext basic_archive_context;
+
+ /*
+ * If we didn't get to storing the pointer to our allocated state, we don't
+ * have anything to clean up.
+ */
+ data = (BasicArchiveData *) (state->private_data);
+ if (data == NULL)
+ return;
+
+ /*
+ * If called within basic_archive_context, switch out of it before we
+ * delete it.
+ */
+ basic_archive_context = data->context;
+ if (CurrentMemoryContext == basic_archive_context)
+ MemoryContextSwitchTo(TopMemoryContext);
+
+ if (MemoryContextIsValid(basic_archive_context))
+ MemoryContextDelete(basic_archive_context);
+
+ /*
+ * Finally, free the state.
+ */
+ pfree(data);
+ state->private_data = NULL;
+}
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..668eb34991 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -125,10 +143,11 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
process exits (e.g., after an error) or the value of
<xref linkend="guc-archive-library"/> changes. If no
<function>shutdown_cb</function> is defined, no special action is taken in
- these situations.
+ these situations. If the archive module has state, this callback should
+ free it to avoid leaks.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..99a07996f7 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -34,8 +34,10 @@
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
+#include "postmaster/archive_module.h"
#include "postmaster/interrupt.h"
#include "postmaster/pgarch.h"
+#include "postmaster/shell_archive.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/latch.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/postmaster/shell_archive.c
index 7771b951b7..54b02ca152 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/postmaster/shell_archive.c
@@ -20,28 +20,34 @@
#include "access/xlog.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
-
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+#include "postmaster/archive_module.h"
+#include "postmaster/shell_archive.h"
+
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..c3ee83b691 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -52,6 +52,7 @@
#include "parser/parse_expr.h"
#include "parser/parser.h"
#include "pgstat.h"
+#include "postmaster/archive_module.h"
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/bgwriter.h"
diff --git a/src/include/postmaster/archive_module.h b/src/include/postmaster/archive_module.h
new file mode 100644
index 0000000000..79e9d53534
--- /dev/null
+++ b/src/include/postmaster/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
diff --git a/src/include/postmaster/shell_archive.h b/src/include/postmaster/shell_archive.h
new file mode 100644
index 0000000000..be83c7715a
--- /dev/null
+++ b/src/include/postmaster/shell_archive.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/postmaster/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+#include "postmaster/archive_module.h"
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
--
2.25.1
--4Ckj6UjgE2iN1+kY--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v11 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 53 ++++++++++++-----
doc/src/sgml/archive-modules.sgml | 32 +++++++---
src/backend/Makefile | 2 +-
src/backend/archive/Makefile | 18 ++++++
src/backend/archive/meson.build | 5 ++
.../{postmaster => archive}/shell_archive.c | 30 ++++++----
src/backend/meson.build | 1 +
src/backend/postmaster/Makefile | 1 -
src/backend/postmaster/meson.build | 1 -
src/backend/postmaster/pgarch.c | 27 +++++----
src/backend/utils/misc/guc_tables.c | 1 +
src/include/archive/archive_module.h | 58 +++++++++++++++++++
src/include/archive/shell_archive.h | 22 +++++++
src/include/postmaster/pgarch.h | 39 -------------
14 files changed, 205 insertions(+), 85 deletions(-)
create mode 100644 src/backend/archive/Makefile
create mode 100644 src/backend/archive/meson.build
rename src/backend/{postmaster => archive}/shell_archive.c (78%)
create mode 100644 src/include/archive/archive_module.h
create mode 100644 src/include/archive/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..8a6bc25db5 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -30,9 +30,9 @@
#include <sys/time.h>
#include <unistd.h>
+#include "archive/archive_module.h"
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,15 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = NULL
+};
+
/*
* _PG_init
*
@@ -67,10 +79,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +86,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +158,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +169,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..8ea697a5bc 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -128,7 +146,7 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
these situations.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 86e6dcb792..e4bf0fe9c0 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -17,7 +17,7 @@ subdir = src/backend
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = access backup bootstrap catalog parser commands executor \
+SUBDIRS = access archive backup bootstrap catalog parser commands executor \
foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
diff --git a/src/backend/archive/Makefile b/src/backend/archive/Makefile
new file mode 100644
index 0000000000..8d2860d0df
--- /dev/null
+++ b/src/backend/archive/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/archive
+#
+# IDENTIFICATION
+# src/backend/archive/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/archive
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ shell_archive.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/archive/meson.build b/src/backend/archive/meson.build
new file mode 100644
index 0000000000..e194282931
--- /dev/null
+++ b/src/backend/archive/meson.build
@@ -0,0 +1,5 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+backend_sources += files(
+ 'shell_archive.c'
+)
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/archive/shell_archive.c
similarity index 78%
rename from src/backend/postmaster/shell_archive.c
rename to src/backend/archive/shell_archive.c
index 7771b951b7..48e8952c0c 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -18,30 +18,36 @@
#include <sys/wait.h>
#include "access/xlog.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/meson.build b/src/backend/meson.build
index b1db3ba75b..4fdd209b82 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -7,6 +7,7 @@ backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
subdir('access')
+subdir('archive')
subdir('backup')
subdir('bootstrap')
subdir('catalog')
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 3a794e54d6..047448b34e 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -22,7 +22,6 @@ OBJS = \
interrupt.o \
pgarch.o \
postmaster.o \
- shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 9079922de7..cda921fd10 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -10,7 +10,6 @@ backend_sources += files(
'interrupt.c',
'pgarch.c',
'postmaster.c',
- 'shell_archive.c',
'startup.c',
'syslogger.c',
'walwriter.c',
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..46af349564 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -31,6 +31,8 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..91cb64602b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -33,6 +33,7 @@
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
#include "access/xlogrecovery.h"
+#include "archive/archive_module.h"
#include "catalog/namespace.h"
#include "catalog/storage.h"
#include "commands/async.h"
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
new file mode 100644
index 0000000000..92ced4b222
--- /dev/null
+++ b/src/include/archive/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/archive/shell_archive.h b/src/include/archive/shell_archive.h
new file mode 100644
index 0000000000..1c843c13c5
--- /dev/null
+++ b/src/include/archive/shell_archive.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
--
2.25.1
--k+w/mQv8wyuph6w0--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v12 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 85 ++++++++++++++++---
doc/src/sgml/archive-modules.sgml | 35 ++++++--
src/backend/Makefile | 2 +-
src/backend/archive/Makefile | 18 ++++
src/backend/archive/meson.build | 5 ++
.../{postmaster => archive}/shell_archive.c | 30 ++++---
src/backend/meson.build | 1 +
src/backend/postmaster/Makefile | 1 -
src/backend/postmaster/meson.build | 1 -
src/backend/postmaster/pgarch.c | 27 +++---
src/backend/utils/misc/guc_tables.c | 1 +
src/include/archive/archive_module.h | 58 +++++++++++++
src/include/archive/shell_archive.h | 22 +++++
src/include/postmaster/pgarch.h | 39 ---------
14 files changed, 239 insertions(+), 86 deletions(-)
create mode 100644 src/backend/archive/Makefile
create mode 100644 src/backend/archive/meson.build
rename src/backend/{postmaster => archive}/shell_archive.c (78%)
create mode 100644 src/include/archive/archive_module.h
create mode 100644 src/include/archive/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..d7c227a10b 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -30,9 +30,9 @@
#include <sys/time.h>
#include <unistd.h>
+#include "archive/archive_module.h"
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,14 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static void basic_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = basic_archive_shutdown
+};
/*
* _PG_init
@@ -67,10 +80,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +87,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +159,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +170,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
@@ -366,3 +394,34 @@ compare_files(const char *file1, const char *file2)
return ret;
}
+
+/*
+ * basic_archive_shutdown
+ *
+ * Frees our allocated state.
+ */
+static void
+basic_archive_shutdown(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context;
+
+ /*
+ * If we didn't get to storing the pointer to our allocated state, we don't
+ * have anything to clean up.
+ */
+ if (data == NULL)
+ return;
+
+ basic_archive_context = data->context;
+ Assert(CurrentMemoryContext != basic_archive_context);
+
+ if (MemoryContextIsValid(basic_archive_context))
+ MemoryContextDelete(basic_archive_context);
+
+ /*
+ * Finally, free the state.
+ */
+ pfree(data);
+ state->private_data = NULL;
+}
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..668eb34991 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -125,10 +143,11 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
process exits (e.g., after an error) or the value of
<xref linkend="guc-archive-library"/> changes. If no
<function>shutdown_cb</function> is defined, no special action is taken in
- these situations.
+ these situations. If the archive module has state, this callback should
+ free it to avoid leaks.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 86e6dcb792..e4bf0fe9c0 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -17,7 +17,7 @@ subdir = src/backend
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = access backup bootstrap catalog parser commands executor \
+SUBDIRS = access archive backup bootstrap catalog parser commands executor \
foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
diff --git a/src/backend/archive/Makefile b/src/backend/archive/Makefile
new file mode 100644
index 0000000000..8d2860d0df
--- /dev/null
+++ b/src/backend/archive/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/archive
+#
+# IDENTIFICATION
+# src/backend/archive/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/archive
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ shell_archive.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/archive/meson.build b/src/backend/archive/meson.build
new file mode 100644
index 0000000000..e194282931
--- /dev/null
+++ b/src/backend/archive/meson.build
@@ -0,0 +1,5 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+backend_sources += files(
+ 'shell_archive.c'
+)
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/archive/shell_archive.c
similarity index 78%
rename from src/backend/postmaster/shell_archive.c
rename to src/backend/archive/shell_archive.c
index 7771b951b7..48e8952c0c 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -18,30 +18,36 @@
#include <sys/wait.h>
#include "access/xlog.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/meson.build b/src/backend/meson.build
index b1db3ba75b..4fdd209b82 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -7,6 +7,7 @@ backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
subdir('access')
+subdir('archive')
subdir('backup')
subdir('bootstrap')
subdir('catalog')
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 3a794e54d6..047448b34e 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -22,7 +22,6 @@ OBJS = \
interrupt.o \
pgarch.o \
postmaster.o \
- shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 9079922de7..cda921fd10 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -10,7 +10,6 @@ backend_sources += files(
'interrupt.c',
'pgarch.c',
'postmaster.c',
- 'shell_archive.c',
'startup.c',
'syslogger.c',
'walwriter.c',
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..46af349564 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -31,6 +31,8 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..91cb64602b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -33,6 +33,7 @@
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
#include "access/xlogrecovery.h"
+#include "archive/archive_module.h"
#include "catalog/namespace.h"
#include "catalog/storage.h"
#include "commands/async.h"
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
new file mode 100644
index 0000000000..92ced4b222
--- /dev/null
+++ b/src/include/archive/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/archive/shell_archive.h b/src/include/archive/shell_archive.h
new file mode 100644
index 0000000000..1c843c13c5
--- /dev/null
+++ b/src/include/archive/shell_archive.h
@@ -0,0 +1,22 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
--
2.25.1
--tThc/1wpZn/ma/RB--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v13 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 85 ++++++++++++++++---
doc/src/sgml/archive-modules.sgml | 35 ++++++--
src/backend/Makefile | 2 +-
src/backend/archive/Makefile | 18 ++++
src/backend/archive/meson.build | 5 ++
.../{postmaster => archive}/shell_archive.c | 30 ++++---
src/backend/meson.build | 1 +
src/backend/postmaster/Makefile | 1 -
src/backend/postmaster/meson.build | 1 -
src/backend/postmaster/pgarch.c | 27 +++---
src/backend/utils/misc/guc_tables.c | 1 +
src/include/archive/archive_module.h | 58 +++++++++++++
src/include/archive/shell_archive.h | 24 ++++++
src/include/postmaster/pgarch.h | 39 ---------
14 files changed, 241 insertions(+), 86 deletions(-)
create mode 100644 src/backend/archive/Makefile
create mode 100644 src/backend/archive/meson.build
rename src/backend/{postmaster => archive}/shell_archive.c (78%)
create mode 100644 src/include/archive/archive_module.h
create mode 100644 src/include/archive/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..d7c227a10b 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -30,9 +30,9 @@
#include <sys/time.h>
#include <unistd.h>
+#include "archive/archive_module.h"
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,14 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static void basic_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = basic_archive_shutdown
+};
/*
* _PG_init
@@ -67,10 +80,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +87,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +159,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +170,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
@@ -366,3 +394,34 @@ compare_files(const char *file1, const char *file2)
return ret;
}
+
+/*
+ * basic_archive_shutdown
+ *
+ * Frees our allocated state.
+ */
+static void
+basic_archive_shutdown(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context;
+
+ /*
+ * If we didn't get to storing the pointer to our allocated state, we don't
+ * have anything to clean up.
+ */
+ if (data == NULL)
+ return;
+
+ basic_archive_context = data->context;
+ Assert(CurrentMemoryContext != basic_archive_context);
+
+ if (MemoryContextIsValid(basic_archive_context))
+ MemoryContextDelete(basic_archive_context);
+
+ /*
+ * Finally, free the state.
+ */
+ pfree(data);
+ state->private_data = NULL;
+}
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..668eb34991 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -125,10 +143,11 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
process exits (e.g., after an error) or the value of
<xref linkend="guc-archive-library"/> changes. If no
<function>shutdown_cb</function> is defined, no special action is taken in
- these situations.
+ these situations. If the archive module has state, this callback should
+ free it to avoid leaks.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 86e6dcb792..e4bf0fe9c0 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -17,7 +17,7 @@ subdir = src/backend
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = access backup bootstrap catalog parser commands executor \
+SUBDIRS = access archive backup bootstrap catalog parser commands executor \
foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
diff --git a/src/backend/archive/Makefile b/src/backend/archive/Makefile
new file mode 100644
index 0000000000..8d2860d0df
--- /dev/null
+++ b/src/backend/archive/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/archive
+#
+# IDENTIFICATION
+# src/backend/archive/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/archive
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ shell_archive.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/archive/meson.build b/src/backend/archive/meson.build
new file mode 100644
index 0000000000..e194282931
--- /dev/null
+++ b/src/backend/archive/meson.build
@@ -0,0 +1,5 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+backend_sources += files(
+ 'shell_archive.c'
+)
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/archive/shell_archive.c
similarity index 78%
rename from src/backend/postmaster/shell_archive.c
rename to src/backend/archive/shell_archive.c
index 7771b951b7..48e8952c0c 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -18,30 +18,36 @@
#include <sys/wait.h>
#include "access/xlog.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/meson.build b/src/backend/meson.build
index b1db3ba75b..4fdd209b82 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -7,6 +7,7 @@ backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
subdir('access')
+subdir('archive')
subdir('backup')
subdir('bootstrap')
subdir('catalog')
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 3a794e54d6..047448b34e 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -22,7 +22,6 @@ OBJS = \
interrupt.o \
pgarch.o \
postmaster.o \
- shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 9079922de7..cda921fd10 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -10,7 +10,6 @@ backend_sources += files(
'interrupt.c',
'pgarch.c',
'postmaster.c',
- 'shell_archive.c',
'startup.c',
'syslogger.c',
'walwriter.c',
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..46af349564 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -31,6 +31,8 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..91cb64602b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -33,6 +33,7 @@
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
#include "access/xlogrecovery.h"
+#include "archive/archive_module.h"
#include "catalog/namespace.h"
#include "catalog/storage.h"
#include "commands/async.h"
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
new file mode 100644
index 0000000000..92ced4b222
--- /dev/null
+++ b/src/include/archive/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/archive/shell_archive.h b/src/include/archive/shell_archive.h
new file mode 100644
index 0000000000..9de6f769f1
--- /dev/null
+++ b/src/include/archive/shell_archive.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+#include "archive/archive_module.h"
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
--
2.25.1
--7AUc2qLy4jB3hD7Z--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v14 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 85 ++++++++++++++++---
doc/src/sgml/archive-modules.sgml | 35 ++++++--
src/backend/Makefile | 2 +-
src/backend/archive/Makefile | 18 ++++
src/backend/archive/meson.build | 5 ++
.../{postmaster => archive}/shell_archive.c | 32 ++++---
src/backend/meson.build | 1 +
src/backend/postmaster/Makefile | 1 -
src/backend/postmaster/meson.build | 1 -
src/backend/postmaster/pgarch.c | 27 +++---
src/backend/utils/misc/guc_tables.c | 1 +
src/include/archive/archive_module.h | 58 +++++++++++++
src/include/archive/shell_archive.h | 24 ++++++
src/include/postmaster/pgarch.h | 39 ---------
14 files changed, 242 insertions(+), 87 deletions(-)
create mode 100644 src/backend/archive/Makefile
create mode 100644 src/backend/archive/meson.build
rename src/backend/{postmaster => archive}/shell_archive.c (77%)
create mode 100644 src/include/archive/archive_module.h
create mode 100644 src/include/archive/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..d7c227a10b 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -30,9 +30,9 @@
#include <sys/time.h>
#include <unistd.h>
+#include "archive/archive_module.h"
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,14 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static void basic_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = basic_archive_shutdown
+};
/*
* _PG_init
@@ -67,10 +80,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +87,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +159,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +170,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
@@ -366,3 +394,34 @@ compare_files(const char *file1, const char *file2)
return ret;
}
+
+/*
+ * basic_archive_shutdown
+ *
+ * Frees our allocated state.
+ */
+static void
+basic_archive_shutdown(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = (BasicArchiveData *) (state->private_data);
+ MemoryContext basic_archive_context;
+
+ /*
+ * If we didn't get to storing the pointer to our allocated state, we don't
+ * have anything to clean up.
+ */
+ if (data == NULL)
+ return;
+
+ basic_archive_context = data->context;
+ Assert(CurrentMemoryContext != basic_archive_context);
+
+ if (MemoryContextIsValid(basic_archive_context))
+ MemoryContextDelete(basic_archive_context);
+
+ /*
+ * Finally, free the state.
+ */
+ pfree(data);
+ state->private_data = NULL;
+}
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..668eb34991 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -125,10 +143,11 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
process exits (e.g., after an error) or the value of
<xref linkend="guc-archive-library"/> changes. If no
<function>shutdown_cb</function> is defined, no special action is taken in
- these situations.
+ these situations. If the archive module has state, this callback should
+ free it to avoid leaks.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 86e6dcb792..e4bf0fe9c0 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -17,7 +17,7 @@ subdir = src/backend
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = access backup bootstrap catalog parser commands executor \
+SUBDIRS = access archive backup bootstrap catalog parser commands executor \
foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
diff --git a/src/backend/archive/Makefile b/src/backend/archive/Makefile
new file mode 100644
index 0000000000..8d2860d0df
--- /dev/null
+++ b/src/backend/archive/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/archive
+#
+# IDENTIFICATION
+# src/backend/archive/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/archive
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ shell_archive.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/archive/meson.build b/src/backend/archive/meson.build
new file mode 100644
index 0000000000..e194282931
--- /dev/null
+++ b/src/backend/archive/meson.build
@@ -0,0 +1,5 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+backend_sources += files(
+ 'shell_archive.c'
+)
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/archive/shell_archive.c
similarity index 77%
rename from src/backend/postmaster/shell_archive.c
rename to src/backend/archive/shell_archive.c
index 7771b951b7..837da543f9 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -9,7 +9,7 @@
* Copyright (c) 2022-2023, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/backend/postmaster/shell_archive.c
+ * src/backend/archive/shell_archive.c
*
*-------------------------------------------------------------------------
*/
@@ -18,30 +18,36 @@
#include <sys/wait.h>
#include "access/xlog.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/meson.build b/src/backend/meson.build
index b1db3ba75b..4fdd209b82 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -7,6 +7,7 @@ backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
subdir('access')
+subdir('archive')
subdir('backup')
subdir('bootstrap')
subdir('catalog')
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 3a794e54d6..047448b34e 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -22,7 +22,6 @@ OBJS = \
interrupt.o \
pgarch.o \
postmaster.o \
- shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 9079922de7..cda921fd10 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -10,7 +10,6 @@ backend_sources += files(
'interrupt.c',
'pgarch.c',
'postmaster.c',
- 'shell_archive.c',
'startup.c',
'syslogger.c',
'walwriter.c',
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..46af349564 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -31,6 +31,8 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 43b9d92660..1c0583fe26 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -33,6 +33,7 @@
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
#include "access/xlogrecovery.h"
+#include "archive/archive_module.h"
#include "catalog/namespace.h"
#include "catalog/storage.h"
#include "commands/async.h"
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
new file mode 100644
index 0000000000..92ced4b222
--- /dev/null
+++ b/src/include/archive/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/archive/shell_archive.h b/src/include/archive/shell_archive.h
new file mode 100644
index 0000000000..9de6f769f1
--- /dev/null
+++ b/src/include/archive/shell_archive.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+#include "archive/archive_module.h"
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
--
2.25.1
--DocE+STaALJfprDB--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v15 1/1] restructure archive modules API
@ 2023-02-09 21:49 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Nathan Bossart @ 2023-02-09 21:49 UTC (permalink / raw)
---
contrib/basic_archive/basic_archive.c | 86 ++++++++++++++++---
doc/src/sgml/archive-modules.sgml | 35 ++++++--
src/backend/Makefile | 2 +-
src/backend/archive/Makefile | 18 ++++
src/backend/archive/meson.build | 5 ++
.../{postmaster => archive}/shell_archive.c | 32 ++++---
src/backend/meson.build | 1 +
src/backend/postmaster/Makefile | 1 -
src/backend/postmaster/meson.build | 1 -
src/backend/postmaster/pgarch.c | 27 +++---
src/backend/utils/misc/guc_tables.c | 1 +
src/include/archive/archive_module.h | 58 +++++++++++++
src/include/archive/shell_archive.h | 24 ++++++
src/include/postmaster/pgarch.h | 39 ---------
14 files changed, 243 insertions(+), 87 deletions(-)
create mode 100644 src/backend/archive/Makefile
create mode 100644 src/backend/archive/meson.build
rename src/backend/{postmaster => archive}/shell_archive.c (77%)
create mode 100644 src/include/archive/archive_module.h
create mode 100644 src/include/archive/shell_archive.h
diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c
index 36b7a4814a..cd852888ce 100644
--- a/contrib/basic_archive/basic_archive.c
+++ b/contrib/basic_archive/basic_archive.c
@@ -30,9 +30,9 @@
#include <sys/time.h>
#include <unistd.h>
+#include "archive/archive_module.h"
#include "common/int.h"
#include "miscadmin.h"
-#include "postmaster/pgarch.h"
#include "storage/copydir.h"
#include "storage/fd.h"
#include "utils/guc.h"
@@ -40,14 +40,27 @@
PG_MODULE_MAGIC;
+typedef struct BasicArchiveData
+{
+ MemoryContext context;
+} BasicArchiveData;
+
static char *archive_directory = NULL;
-static MemoryContext basic_archive_context;
-static bool basic_archive_configured(void);
-static bool basic_archive_file(const char *file, const char *path);
+static void basic_archive_startup(ArchiveModuleState *state);
+static bool basic_archive_configured(ArchiveModuleState *state);
+static bool basic_archive_file(ArchiveModuleState *state, const char *file, const char *path);
static void basic_archive_file_internal(const char *file, const char *path);
static bool check_archive_directory(char **newval, void **extra, GucSource source);
static bool compare_files(const char *file1, const char *file2);
+static void basic_archive_shutdown(ArchiveModuleState *state);
+
+static const ArchiveModuleCallbacks basic_archive_callbacks = {
+ .startup_cb = basic_archive_startup,
+ .check_configured_cb = basic_archive_configured,
+ .archive_file_cb = basic_archive_file,
+ .shutdown_cb = basic_archive_shutdown
+};
/*
* _PG_init
@@ -67,10 +80,6 @@ _PG_init(void)
check_archive_directory, NULL, NULL);
MarkGUCPrefixReserved("basic_archive");
-
- basic_archive_context = AllocSetContextCreate(TopMemoryContext,
- "basic_archive",
- ALLOCSET_DEFAULT_SIZES);
}
/*
@@ -78,11 +87,28 @@ _PG_init(void)
*
* Returns the module's archiving callbacks.
*/
+const ArchiveModuleCallbacks *
+_PG_archive_module_init(void)
+{
+ return &basic_archive_callbacks;
+}
+
+/*
+ * basic_archive_startup
+ *
+ * Creates the module's memory context.
+ */
void
-_PG_archive_module_init(ArchiveModuleCallbacks *cb)
+basic_archive_startup(ArchiveModuleState *state)
{
- cb->check_configured_cb = basic_archive_configured;
- cb->archive_file_cb = basic_archive_file;
+ BasicArchiveData *data;
+
+ data = (BasicArchiveData *) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(BasicArchiveData));
+ data->context = AllocSetContextCreate(TopMemoryContext,
+ "basic_archive",
+ ALLOCSET_DEFAULT_SIZES);
+ state->private_data = (void *) data;
}
/*
@@ -133,7 +159,7 @@ check_archive_directory(char **newval, void **extra, GucSource source)
* Checks that archive_directory is not blank.
*/
static bool
-basic_archive_configured(void)
+basic_archive_configured(ArchiveModuleState *state)
{
return archive_directory != NULL && archive_directory[0] != '\0';
}
@@ -144,10 +170,12 @@ basic_archive_configured(void)
* Archives one file.
*/
static bool
-basic_archive_file(const char *file, const char *path)
+basic_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
sigjmp_buf local_sigjmp_buf;
MemoryContext oldcontext;
+ BasicArchiveData *data = (BasicArchiveData *) state->private_data;
+ MemoryContext basic_archive_context = data->context;
/*
* We run basic_archive_file_internal() in our own memory context so that
@@ -366,3 +394,35 @@ compare_files(const char *file1, const char *file2)
return ret;
}
+
+/*
+ * basic_archive_shutdown
+ *
+ * Frees our allocated state.
+ */
+static void
+basic_archive_shutdown(ArchiveModuleState *state)
+{
+ BasicArchiveData *data = (BasicArchiveData *) state->private_data;
+ MemoryContext basic_archive_context;
+
+ /*
+ * If we didn't get to storing the pointer to our allocated state, we don't
+ * have anything to clean up.
+ */
+ if (data == NULL)
+ return;
+
+ basic_archive_context = data->context;
+ Assert(CurrentMemoryContext != basic_archive_context);
+
+ if (MemoryContextIsValid(basic_archive_context))
+ MemoryContextDelete(basic_archive_context);
+ data->context = NULL;
+
+ /*
+ * Finally, free the state.
+ */
+ pfree(data);
+ state->private_data = NULL;
+}
diff --git a/doc/src/sgml/archive-modules.sgml b/doc/src/sgml/archive-modules.sgml
index ef02051f7f..668eb34991 100644
--- a/doc/src/sgml/archive-modules.sgml
+++ b/doc/src/sgml/archive-modules.sgml
@@ -47,18 +47,22 @@
normal library search path is used to locate the library. To provide the
required archive module callbacks and to indicate that the library is
actually an archive module, it needs to provide a function named
- <function>_PG_archive_module_init</function>. This function is passed a
- struct that needs to be filled with the callback function pointers for
- individual actions.
+ <function>_PG_archive_module_init</function>. The result of the function
+ must be a pointer to a struct of type
+ <structname>ArchiveModuleCallbacks</structname>, which contains everything
+ that the core code needs to know how to make use of the archive module. The
+ return value needs to be of server lifetime, which is typically achieved by
+ defining it as a <literal>static const</literal> variable in global scope.
<programlisting>
typedef struct ArchiveModuleCallbacks
{
+ ArchiveStartupCB startup_cb;
ArchiveCheckConfiguredCB check_configured_cb;
ArchiveFileCB archive_file_cb;
ArchiveShutdownCB shutdown_cb;
} ArchiveModuleCallbacks;
-typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
</programlisting>
Only the <function>archive_file_cb</function> callback is required. The
@@ -73,6 +77,20 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
The server will call them as required to process each individual WAL file.
</para>
+ <sect2 id="archive-module-startup">
+ <title>Startup Callback</title>
+ <para>
+ The <function>startup_cb</function> callback is called shortly after the
+ module is loaded. This callback can be used to perform any additional
+ initialization required. If the archive module has state, it can use
+ <structfield>state->private_data</structfield> to store it.
+
+<programlisting>
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+</programlisting>
+ </para>
+ </sect2>
+
<sect2 id="archive-module-check">
<title>Check Callback</title>
<para>
@@ -83,7 +101,7 @@ typedef void (*ArchiveModuleInit) (struct ArchiveModuleCallbacks *cb);
assumes the module is configured.
<programlisting>
-typedef bool (*ArchiveCheckConfiguredCB) (void);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
</programlisting>
If <literal>true</literal> is returned, the server will proceed with
@@ -105,7 +123,7 @@ WARNING: archive_mode enabled, yet archiving is not configured
single WAL file.
<programlisting>
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
</programlisting>
If <literal>true</literal> is returned, the server proceeds as if the file
@@ -125,10 +143,11 @@ typedef bool (*ArchiveFileCB) (const char *file, const char *path);
process exits (e.g., after an error) or the value of
<xref linkend="guc-archive-library"/> changes. If no
<function>shutdown_cb</function> is defined, no special action is taken in
- these situations.
+ these situations. If the archive module has state, this callback should
+ free it to avoid leaks.
<programlisting>
-typedef void (*ArchiveShutdownCB) (void);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
</programlisting>
</para>
</sect2>
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 86e6dcb792..e4bf0fe9c0 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -17,7 +17,7 @@ subdir = src/backend
top_builddir = ../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = access backup bootstrap catalog parser commands executor \
+SUBDIRS = access archive backup bootstrap catalog parser commands executor \
foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
diff --git a/src/backend/archive/Makefile b/src/backend/archive/Makefile
new file mode 100644
index 0000000000..8d2860d0df
--- /dev/null
+++ b/src/backend/archive/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for src/backend/archive
+#
+# IDENTIFICATION
+# src/backend/archive/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/archive
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ shell_archive.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/archive/meson.build b/src/backend/archive/meson.build
new file mode 100644
index 0000000000..e194282931
--- /dev/null
+++ b/src/backend/archive/meson.build
@@ -0,0 +1,5 @@
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+backend_sources += files(
+ 'shell_archive.c'
+)
diff --git a/src/backend/postmaster/shell_archive.c b/src/backend/archive/shell_archive.c
similarity index 77%
rename from src/backend/postmaster/shell_archive.c
rename to src/backend/archive/shell_archive.c
index 7771b951b7..837da543f9 100644
--- a/src/backend/postmaster/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -9,7 +9,7 @@
* Copyright (c) 2022-2023, PostgreSQL Global Development Group
*
* IDENTIFICATION
- * src/backend/postmaster/shell_archive.c
+ * src/backend/archive/shell_archive.c
*
*-------------------------------------------------------------------------
*/
@@ -18,30 +18,36 @@
#include <sys/wait.h>
#include "access/xlog.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "common/percentrepl.h"
#include "pgstat.h"
-#include "postmaster/pgarch.h"
-static bool shell_archive_configured(void);
-static bool shell_archive_file(const char *file, const char *path);
-static void shell_archive_shutdown(void);
+static bool shell_archive_configured(ArchiveModuleState *state);
+static bool shell_archive_file(ArchiveModuleState *state, const char *file, const char *path);
+static void shell_archive_shutdown(ArchiveModuleState *state);
-void
-shell_archive_init(ArchiveModuleCallbacks *cb)
+static const ArchiveModuleCallbacks shell_archive_callbacks = {
+ .startup_cb = NULL,
+ .check_configured_cb = shell_archive_configured,
+ .archive_file_cb = shell_archive_file,
+ .shutdown_cb = shell_archive_shutdown
+};
+
+const ArchiveModuleCallbacks *
+shell_archive_init(void)
{
- cb->check_configured_cb = shell_archive_configured;
- cb->archive_file_cb = shell_archive_file;
- cb->shutdown_cb = shell_archive_shutdown;
+ return &shell_archive_callbacks;
}
static bool
-shell_archive_configured(void)
+shell_archive_configured(ArchiveModuleState *state)
{
return XLogArchiveCommand[0] != '\0';
}
static bool
-shell_archive_file(const char *file, const char *path)
+shell_archive_file(ArchiveModuleState *state, const char *file, const char *path)
{
char *xlogarchcmd;
char *nativePath = NULL;
@@ -123,7 +129,7 @@ shell_archive_file(const char *file, const char *path)
}
static void
-shell_archive_shutdown(void)
+shell_archive_shutdown(ArchiveModuleState *state)
{
elog(DEBUG1, "archiver process shutting down");
}
diff --git a/src/backend/meson.build b/src/backend/meson.build
index b1db3ba75b..4fdd209b82 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -7,6 +7,7 @@ backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
subdir('access')
+subdir('archive')
subdir('backup')
subdir('bootstrap')
subdir('catalog')
diff --git a/src/backend/postmaster/Makefile b/src/backend/postmaster/Makefile
index 3a794e54d6..047448b34e 100644
--- a/src/backend/postmaster/Makefile
+++ b/src/backend/postmaster/Makefile
@@ -22,7 +22,6 @@ OBJS = \
interrupt.o \
pgarch.o \
postmaster.o \
- shell_archive.o \
startup.o \
syslogger.o \
walwriter.o
diff --git a/src/backend/postmaster/meson.build b/src/backend/postmaster/meson.build
index 9079922de7..cda921fd10 100644
--- a/src/backend/postmaster/meson.build
+++ b/src/backend/postmaster/meson.build
@@ -10,7 +10,6 @@ backend_sources += files(
'interrupt.c',
'pgarch.c',
'postmaster.c',
- 'shell_archive.c',
'startup.c',
'syslogger.c',
'walwriter.c',
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index e551af2905..46af349564 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -31,6 +31,8 @@
#include "access/xlog.h"
#include "access/xlog_internal.h"
+#include "archive/archive_module.h"
+#include "archive/shell_archive.h"
#include "lib/binaryheap.h"
#include "libpq/pqsignal.h"
#include "pgstat.h"
@@ -97,7 +99,8 @@ char *XLogArchiveLibrary = "";
*/
static time_t last_sigterm_time = 0;
static PgArchData *PgArch = NULL;
-static ArchiveModuleCallbacks ArchiveContext;
+static const ArchiveModuleCallbacks *ArchiveCallbacks;
+static ArchiveModuleState *archive_module_state;
/*
@@ -406,8 +409,8 @@ pgarch_ArchiverCopyLoop(void)
HandlePgArchInterrupts();
/* can't do anything if not configured ... */
- if (ArchiveContext.check_configured_cb != NULL &&
- !ArchiveContext.check_configured_cb())
+ if (ArchiveCallbacks->check_configured_cb != NULL &&
+ !ArchiveCallbacks->check_configured_cb(archive_module_state))
{
ereport(WARNING,
(errmsg("archive_mode enabled, yet archiving is not configured")));
@@ -508,7 +511,7 @@ pgarch_archiveXlog(char *xlog)
snprintf(activitymsg, sizeof(activitymsg), "archiving %s", xlog);
set_ps_display(activitymsg);
- ret = ArchiveContext.archive_file_cb(xlog, pathname);
+ ret = ArchiveCallbacks->archive_file_cb(archive_module_state, xlog, pathname);
if (ret)
snprintf(activitymsg, sizeof(activitymsg), "last was %s", xlog);
else
@@ -814,7 +817,7 @@ HandlePgArchInterrupts(void)
/*
* LoadArchiveLibrary
*
- * Loads the archiving callbacks into our local ArchiveContext.
+ * Loads the archiving callbacks into our local ArchiveCallbacks.
*/
static void
LoadArchiveLibrary(void)
@@ -827,8 +830,6 @@ LoadArchiveLibrary(void)
errmsg("both archive_command and archive_library set"),
errdetail("Only one of archive_command, archive_library may be set.")));
- memset(&ArchiveContext, 0, sizeof(ArchiveModuleCallbacks));
-
/*
* If shell archiving is enabled, use our special initialization function.
* Otherwise, load the library and call its _PG_archive_module_init().
@@ -844,12 +845,16 @@ LoadArchiveLibrary(void)
ereport(ERROR,
(errmsg("archive modules have to define the symbol %s", "_PG_archive_module_init")));
- (*archive_init) (&ArchiveContext);
+ ArchiveCallbacks = (*archive_init) ();
- if (ArchiveContext.archive_file_cb == NULL)
+ if (ArchiveCallbacks->archive_file_cb == NULL)
ereport(ERROR,
(errmsg("archive modules must register an archive callback")));
+ archive_module_state = (ArchiveModuleState *) palloc0(sizeof(ArchiveModuleState));
+ if (ArchiveCallbacks->startup_cb != NULL)
+ ArchiveCallbacks->startup_cb(archive_module_state);
+
before_shmem_exit(pgarch_call_module_shutdown_cb, 0);
}
@@ -859,6 +864,6 @@ LoadArchiveLibrary(void)
static void
pgarch_call_module_shutdown_cb(int code, Datum arg)
{
- if (ArchiveContext.shutdown_cb != NULL)
- ArchiveContext.shutdown_cb();
+ if (ArchiveCallbacks->shutdown_cb != NULL)
+ ArchiveCallbacks->shutdown_cb(archive_module_state);
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 43b9d92660..1c0583fe26 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -33,6 +33,7 @@
#include "access/xlog_internal.h"
#include "access/xlogprefetcher.h"
#include "access/xlogrecovery.h"
+#include "archive/archive_module.h"
#include "catalog/namespace.h"
#include "catalog/storage.h"
#include "commands/async.h"
diff --git a/src/include/archive/archive_module.h b/src/include/archive/archive_module.h
new file mode 100644
index 0000000000..92ced4b222
--- /dev/null
+++ b/src/include/archive/archive_module.h
@@ -0,0 +1,58 @@
+/*-------------------------------------------------------------------------
+ *
+ * archive_module.h
+ * Exports for archive modules.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/archive_module.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _ARCHIVE_MODULE_H
+#define _ARCHIVE_MODULE_H
+
+/*
+ * The value of the archive_library GUC.
+ */
+extern PGDLLIMPORT char *XLogArchiveLibrary;
+
+typedef struct ArchiveModuleState
+{
+ /*
+ * Private data pointer for use by an archive module. This can be used to
+ * store state for the module that will be passed to each of its callbacks.
+ */
+ void *private_data;
+} ArchiveModuleState;
+
+/*
+ * Archive module callbacks
+ *
+ * These callback functions should be defined by archive libraries and returned
+ * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
+ * For more information about the purpose of each callback, refer to the
+ * archive modules documentation.
+ */
+typedef void (*ArchiveStartupCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state);
+typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path);
+typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state);
+
+typedef struct ArchiveModuleCallbacks
+{
+ ArchiveStartupCB startup_cb;
+ ArchiveCheckConfiguredCB check_configured_cb;
+ ArchiveFileCB archive_file_cb;
+ ArchiveShutdownCB shutdown_cb;
+} ArchiveModuleCallbacks;
+
+/*
+ * Type of the shared library symbol _PG_archive_module_init that is looked
+ * up when loading an archive library.
+ */
+typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void);
+
+extern PGDLLEXPORT const ArchiveModuleCallbacks *_PG_archive_module_init(void);
+
+#endif /* _ARCHIVE_MODULE_H */
diff --git a/src/include/archive/shell_archive.h b/src/include/archive/shell_archive.h
new file mode 100644
index 0000000000..9de6f769f1
--- /dev/null
+++ b/src/include/archive/shell_archive.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * shell_archive.h
+ * Exports for archiving via shell.
+ *
+ * Copyright (c) 2022-2023, PostgreSQL Global Development Group
+ *
+ * src/include/archive/shell_archive.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef _SHELL_ARCHIVE_H
+#define _SHELL_ARCHIVE_H
+
+#include "archive/archive_module.h"
+
+/*
+ * Since the logic for archiving via a shell command is in the core server
+ * and does not need to be loaded via a shared library, it has a special
+ * initialization function.
+ */
+extern const ArchiveModuleCallbacks *shell_archive_init(void);
+
+#endif /* _SHELL_ARCHIVE_H */
diff --git a/src/include/postmaster/pgarch.h b/src/include/postmaster/pgarch.h
index bcd51dfad6..3bd4fac71e 100644
--- a/src/include/postmaster/pgarch.h
+++ b/src/include/postmaster/pgarch.h
@@ -33,43 +33,4 @@ extern void PgArchiverMain(void) pg_attribute_noreturn();
extern void PgArchWakeup(void);
extern void PgArchForceDirScan(void);
-/*
- * The value of the archive_library GUC.
- */
-extern PGDLLIMPORT char *XLogArchiveLibrary;
-
-/*
- * Archive module callbacks
- *
- * These callback functions should be defined by archive libraries and returned
- * via _PG_archive_module_init(). ArchiveFileCB is the only required callback.
- * For more information about the purpose of each callback, refer to the
- * archive modules documentation.
- */
-typedef bool (*ArchiveCheckConfiguredCB) (void);
-typedef bool (*ArchiveFileCB) (const char *file, const char *path);
-typedef void (*ArchiveShutdownCB) (void);
-
-typedef struct ArchiveModuleCallbacks
-{
- ArchiveCheckConfiguredCB check_configured_cb;
- ArchiveFileCB archive_file_cb;
- ArchiveShutdownCB shutdown_cb;
-} ArchiveModuleCallbacks;
-
-/*
- * Type of the shared library symbol _PG_archive_module_init that is looked
- * up when loading an archive library.
- */
-typedef void (*ArchiveModuleInit) (ArchiveModuleCallbacks *cb);
-
-extern PGDLLEXPORT void _PG_archive_module_init(ArchiveModuleCallbacks *cb);
-
-/*
- * Since the logic for archiving via a shell command is in the core server
- * and does not need to be loaded via a shared library, it has a special
- * initialization function.
- */
-extern void shell_archive_init(ArchiveModuleCallbacks *cb);
-
#endif /* _PGARCH_H */
--
2.25.1
--HcAYCG3uE/tztfnV--
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: zlib detection in Meson on Windows broken?
@ 2024-05-21 14:12 Sandeep Thakkar <[email protected]>
0 siblings, 1 reply; 20+ messages in thread
From: Sandeep Thakkar @ 2024-05-21 14:12 UTC (permalink / raw)
To: Dave Page <[email protected]>; Muralikrishna Bandaru <[email protected]>; +Cc: Nazir Bilal Yavuz <[email protected]>; PostgreSQL Developers <[email protected]>
Hi Dave,
On Tue, May 21, 2024 at 3:12 PM Dave Page <[email protected]> wrote:
> Hi Sandeep, Nazir,
>
> On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz <[email protected]>
> wrote:
>
>> Hi,
>>
>> On Tue, 21 May 2024 at 10:20, Sandeep Thakkar
>> <[email protected]> wrote:
>> >
>> > Hi Dave,
>> >
>> > Is the .pc file generated after the successful build of zlib? If yes,
>> then meson should be able to detect the installation ideally
>>
>> If meson is not able to find the .pc file automatically, using 'meson
>> setup ... --pkg-config-path $ZLIB_PC_PATH' might help.
>>
>
> The problem is that on Windows there are no standard locations for a
> Unix-style development library installation such as this, so the chances
> are that the .pc file will point to entirely the wrong location.
>
> For example, please see
> https://github.com/dpage/winpgbuild/actions/runs/9172187335 which is a
> Github action that builds a completely vanilla zlib using VC++. If you look
> at the uploaded artefact containing the build output and example the .pc
> file, you'll see it references /zlib as the location, which is simply where
> I built it in that action. On a developer's machine that's almost certainly
> not going to be where it actually ends up. For example, on the pgAdmin
> build farm, the dependencies all end up in C:\build64\[whatever]. On the
> similar Github action I'm building for PostgreSQL, that artefact will be
> unpacked into /build/zlib.
>
>
The above link returned 404. But I found a successful build at
https://github.com/dpage/winpgbuild/actions/runs/9175426807. I downloaded
the artifact but didn't find .pc file as I wanted to look into the content
of that file.
I had a word with Murali who mentioned he encountered a similar issue while
building PG17 on windows. He worked-around is by using a template .pc file
that includes these lines:
--
prefix=${pcfiledir}/../..
exec_prefix=${prefix}
libdir=${prefix}/lib
sharedlibdir=${prefix}/lib
includedir=${prefix}/include
--
But in general I agree with you on the issue of Meson's dependency on
pkgconfig files to detect the third party libraries.
Of course, for my own builds I can easily make everything use consistent
> directories, however most people who are likely to want to build PostgreSQL
> may not want to also build all the dependencies themselves as well, as some
> are a lot more difficult than zlib. So what tends to happen is people find
> third party builds or upstream official builds.
>
> I would therefore argue that if the .pc file that's found doesn't provide
> correct paths for us, then Meson should fall back to searching in the paths
> specified on its command line for the appropriate libraries/headers (which
> is what it does for OpenSSL for example, as that doesn't include a .pc
> file). This is also what happens with PG16 and earlier.
>
> One other thing I will note is that PG16 and earlier try to use the wrong
> filename for the import library. For years, it's been a requirement to do
> something like this: "copy \zlib\lib\zlib.lib \zlib\lib\zdll.lib" to make a
> build succeed against a "vanilla" zlib build. I haven't got as far as
> figuring out if the same is true with Meson yet.
>
> --
> Dave Page
> pgAdmin: https://www.pgadmin.org
> PostgreSQL: https://www.postgresql.org
> EDB: https://www.enterprisedb.com
>
>
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: zlib detection in Meson on Windows broken?
@ 2024-05-21 14:24 Dave Page <[email protected]>
parent: Sandeep Thakkar <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Dave Page @ 2024-05-21 14:24 UTC (permalink / raw)
To: Sandeep Thakkar <[email protected]>; +Cc: Muralikrishna Bandaru <[email protected]>; Nazir Bilal Yavuz <[email protected]>; PostgreSQL Developers <[email protected]>
Hi
On Tue, 21 May 2024 at 15:12, Sandeep Thakkar <
[email protected]> wrote:
> Hi Dave,
>
>
> On Tue, May 21, 2024 at 3:12 PM Dave Page <[email protected]> wrote:
>
>> Hi Sandeep, Nazir,
>>
>> On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> On Tue, 21 May 2024 at 10:20, Sandeep Thakkar
>>> <[email protected]> wrote:
>>> >
>>> > Hi Dave,
>>> >
>>> > Is the .pc file generated after the successful build of zlib? If yes,
>>> then meson should be able to detect the installation ideally
>>>
>>> If meson is not able to find the .pc file automatically, using 'meson
>>> setup ... --pkg-config-path $ZLIB_PC_PATH' might help.
>>>
>>
>> The problem is that on Windows there are no standard locations for a
>> Unix-style development library installation such as this, so the chances
>> are that the .pc file will point to entirely the wrong location.
>>
>> For example, please see
>> https://github.com/dpage/winpgbuild/actions/runs/9172187335 which is a
>> Github action that builds a completely vanilla zlib using VC++. If you look
>> at the uploaded artefact containing the build output and example the .pc
>> file, you'll see it references /zlib as the location, which is simply where
>> I built it in that action. On a developer's machine that's almost certainly
>> not going to be where it actually ends up. For example, on the pgAdmin
>> build farm, the dependencies all end up in C:\build64\[whatever]. On the
>> similar Github action I'm building for PostgreSQL, that artefact will be
>> unpacked into /build/zlib.
>>
>>
> The above link returned 404. But I found a successful build at
> https://github.com/dpage/winpgbuild/actions/runs/9175426807. I downloaded
> the artifact but didn't find .pc file as I wanted to look into the content
> of that file.
>
Yeah, sorry - that was an old one.
Following some offline discussion with Andrew I realised I was building
zlib incorrectly - using cmake/msbuild instead of nmake and some manual
copying. Whilst the former does create a working library and a sane looking
installation, it's not the recommended method, which is documented in a
very non-obvious way - far less obvious than "oh look, there's a cmake
config - let's use that".
The new build is done using the recommended method, which works with PG16
and below out of the box with no need to rename any files.
>
> I had a word with Murali who mentioned he encountered a similar issue
> while building PG17 on windows. He worked-around is by using a template .pc
> file that includes these lines:
> --
> prefix=${pcfiledir}/../..
> exec_prefix=${prefix}
> libdir=${prefix}/lib
> sharedlibdir=${prefix}/lib
> includedir=${prefix}/include
> --
>
The issue here is that there is no .pc file created with the correct way of
building zlib, and even if there were (or I created a dummy one),
pkg-config isn't really a thing on Windows.
I'd also note that from what Andrew has shown me of the zlib installation
on the buildfarm member drongo, there is no .pc file there either, and yet
seems to work fine (and my zlib installation now has the exact same set of
files as his does).
>
> But in general I agree with you on the issue of Meson's dependency on
> pkgconfig files to detect the third party libraries.
>
> Of course, for my own builds I can easily make everything use consistent
>> directories, however most people who are likely to want to build PostgreSQL
>> may not want to also build all the dependencies themselves as well, as some
>> are a lot more difficult than zlib. So what tends to happen is people find
>> third party builds or upstream official builds.
>>
>> I would therefore argue that if the .pc file that's found doesn't provide
>> correct paths for us, then Meson should fall back to searching in the paths
>> specified on its command line for the appropriate libraries/headers (which
>> is what it does for OpenSSL for example, as that doesn't include a .pc
>> file). This is also what happens with PG16 and earlier.
>>
>> One other thing I will note is that PG16 and earlier try to use the wrong
>> filename for the import library. For years, it's been a requirement to do
>> something like this: "copy \zlib\lib\zlib.lib \zlib\lib\zdll.lib" to make a
>> build succeed against a "vanilla" zlib build. I haven't got as far as
>> figuring out if the same is true with Meson yet.
>>
>> --
>> Dave Page
>> pgAdmin: https://www.pgadmin.org
>> PostgreSQL: https://www.postgresql.org
>> EDB: https://www.enterprisedb.com
>>
>>
>
> --
> Sandeep Thakkar
>
>
>
--
Dave Page
pgAdmin: https://www.pgadmin.org
PostgreSQL: https://www.postgresql.org
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v22 5/8] Row pattern recognition patch (executor).
@ 2024-09-19 04:48 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 51a6708a39..e46a3dd1b7 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1199,6 +1353,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2170,6 +2325,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2278,6 +2438,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2445,6 +2616,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2543,6 +2717,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2724,6 +2908,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2732,6 +2953,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2789,6 +3068,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3149,7 +3430,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3469,14 +3751,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3543,11 +3865,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3614,6 +3950,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3632,15 +3976,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3671,3 +4013,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 43f608d7a0..6a301920f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10604,6 +10604,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 88467977f8..058cef2121 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2581,6 +2581,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2640,6 +2645,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2667,6 +2685,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Thu_Sep_19_13_59_47_2024_608)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v22-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 20+ messages in thread
end of thread, other threads:[~2024-09-19 04:48 UTC | newest]
Thread overview: 20+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH 3/6] Add primary_slot_name to init_from_backup in TAP test. Kyotaro Horiguchi <[email protected]>
2019-03-20 00:26 [PATCH 1/2] Add IntegerSet, to hold large sets of 64-bit ints efficiently. Heikki Linnakangas <[email protected]>
2023-01-31 22:58 [PATCH v8 4/4] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v9 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v4 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v5 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v6 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v7 3/4] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v3 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-01-31 22:58 [PATCH v2 3/3] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v11 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v13 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v10 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v14 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v12 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2023-02-09 21:49 [PATCH v15 1/1] restructure archive modules API Nathan Bossart <[email protected]>
2024-05-21 14:12 Re: zlib detection in Meson on Windows broken? Sandeep Thakkar <[email protected]>
2024-05-21 14:24 ` Re: zlib detection in Meson on Windows broken? Dave Page <[email protected]>
2024-09-19 04:48 [PATCH v22 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox